Fix LoRA scaling: divide alpha by rank (#845)

[H-A-Khan]

What

Apply the standard LoRA scaling factor alpha / rank (Hu et al. 2021) in LoRaLayer.__call__ and replace_lora_with_linear. Previously the layer multiplied the LoRA update by raw alpha, making the effective scaling rank-times too large for the documented defaults — for example r=8, alpha=16 gave an effective scaling of 16 instead of the intended 2.

This matches every other PEFT implementation, including HuggingFace peft and the original Microsoft LoRA reference.

Closes #845.

Changes

• LoRaLayer.__init__ now stores self.rank and self.scaling = alpha / rank.
• LoRaLayer.__call__ multiplies the update by self.scaling.
• replace_lora_with_linear uses layer.scaling so merged weights match what the trained adapter applies during inference.
• Two regression tests in mlx_vlm/tests/test_trainer_utils.py:
  • test_lora_layer_uses_alpha_over_rank_scaling — checks the stored attribute.
  • test_lora_layer_forward_matches_alpha_over_rank — checks the actual forward-pass output against the expected base + (alpha/rank) * (x A B) formula with a non-zero B.

Backwards compatibility

⚠️ This is a behavioural change. Adapters trained against the previous (broken) scaling will behave 8× weaker after this fix when r=8, alpha=16. Users who want the old effective scaling can multiply their alpha by rank — e.g. set --lora-alpha 128 to recover the old r=8, alpha=16 behaviour. The recommended action is to retrain with the documented defaults now that they actually mean what the docs say.

If maintainers prefer to gate this behind a flag for one release I am happy to do that — let me know.

Tests

python -m pytest mlx_vlm/tests/test_trainer_utils.py -v

test_find_all_linear_names PASSED
test_get_module_by_name PASSED
test_get_peft_model PASSED
test_lora_layer_forward_matches_alpha_over_rank PASSED
test_lora_layer_uses_alpha_over_rank_scaling PASSED
test_set_module_by_name PASSED
======================== 6 passed in 4.97s =========================