LoRA

How it works

For a weight matrix W of size d x k, LoRA learns two small matrices, B (d x r) and A (r x k), with a small rank r such as 8 to 64, and uses W + BA in place of W. Only A and B are trained, so trainable parameters and optimizer memory drop sharply. QLoRA trains LoRA adapters on top of a 4-bit quantized base to save more memory.

Adapters are small files that can be merged into the weights or kept separate, and engines such as vLLM can serve many adapters on one base model at once. Providers' fine-tuning APIs may use parameter-efficient methods behind the scenes without saying which.

LoRA often comes close to full fine-tuning for style, format and narrow tasks, but adds less new knowledge. An adapter only works with the exact base checkpoint it was trained on.

Worked example

Illustrative: for a 4,096 x 4,096 matrix, full fine-tuning updates 4,096 x 4,096 = 16,777,216 parameters. LoRA with rank 16 trains 4,096 x 16 + 16 x 4,096 = 131,072, about 0.8% as many.

Common mistakes

• 1Loading an adapter on a different checkpoint or quantization than it was trained on.
• 2Expecting LoRA to teach a large body of new facts; it mainly adjusts behaviour and format.
• 3Skipping evaluation after merging or quantizing an adapter, which can change results.