LoRA: Low-Rank Adaptation

From the previous two topics, in two plain sentences: full fine-tuning updates every weight and pays ~16 bytes of memory per parameter (topic 175); PEFT freezes the base and trains a tiny slice because task adaptation is low-dimensional (topic 176).

LoRA is the specific way PEFT exploits that last word. Look at what "the task needs only a small change" means on one layer. A Transformer attention block has weight matrices like W₀ of shape 4096×4096 — 16.7 million numbers. Fine-tuning it fully means giving all 16.7M numbers their own gradients and AdamW backpacks, to represent… how much actual new behavior?

The measurements (topic 176's Aghajanyan result, extended by the LoRA paper itself): the delta ΔW between base and fine-tuned weights is thin — it can be approximated by a matrix of rank 4–16 with almost no quality lost. In plain words: the change fits in a handful of directions, yet plain fine-tuning spends machinery describing all 16.7M coordinates of it, most of them ~zero.

So the question becomes

Hu et al. (2021, arXiv:2106.09685) answered, and their answer became the de facto standard of LLM adaptation.

LoRA is one displayed equation and one hypothesis:

Unpack the pieces:

• W₀ — the pretrained weight matrix. Frozen: requires_grad=False. Gradients never flow into it; it never moves.
• ΔW — the change LoRA actually learns, never stored as a full d×k matrix. Instead it is a product of two skinny matrices:
  • A (r×k) — the down-projection: squeezes the input from k dimensions into an r-dimensional bottleneck. Random Gaussian-initialized (to break symmetry).
  • B (d×r) — the up-projection: expands the bottleneck back to d dimensions. Initialized to all zeros.
• r — the rank: how many directions of change you allow. The bottleneck width.
• α/r (γ) — a fixed scaling constant that lets you set the adapter's strength without touching the learning rate.

The hypothesis behind the design: ΔW has small intrinsic rank — the task delta lives in a low-dimensional subspace of the enormous pretrained parameter space. So don't learn 16.7M numbers; learn r·(d+k) numbers that reconstruct a rank-r approximation of the delta.

Three properties made this the industry default:

1. Zero-init. B starts at zero, so ΔW = BA = 0 at step 0. Training begins exactly at the pretrained model — there is no random perturbation for the first steps to undo. Stability, for free.
2. No inference latency. BA multiplies through associatively; at export you compute W′ = W₀ + (α/r)BA and ship a plain model. Nothing extra runs at serve time.
3. Enormous compression. On GPT-3 175B, LoRA trained ~0.01% of parameters (~35M at rank 4 in the paper's setup; the headline demo used 350M vs 175B). Full fine-tuning would need ~700 GB of fp32 optimizer states plus the model; the paper cut GPU memory ~3x and wall-clock step time ~25% versus the same model with adapters-in-the-path.

Take one projection matrix: W₀ is 4096×4096 = 16,777,216 parameters.

Full fine-tuning trains all of them:

trainable: 16.7M          (AdamW backpack: 16.7M × 12 ≈ 201 MB more)

LoRA with rank r = 16:

code
Copy
A: 16 × 4096  =   65,536
B: 4096 × 16  =   65,536
ΔW trainable:    131,072   = r·(d+k) = 16 × 8192
                ≈ 0.78% of the matrix it approximates

Two knobs to make the numbers concrete:

• Why is a 131k-param product enough to represent a 16.7M matrix's change? Because a rank-16 product can only bend the layer along 16 directions — and the measurements say task updates are that bendy. You're trading expressiveness for the empirically true assumption "the delta is thin." (One plain sentence on rank: the number of independent directions a matrix actually moves things in; most fine-tune deltas need fewer than 16.)
• γ = α/r: with the community default r=16, α=32, every adapter output is scaled by γ = 2 — α is just a gain knob bolted onto the branch.

Do the same count for the whole model and you get the receipt from topic 176's code: a 7B model with rank-16 on all linear layers → ~20M trainable, ~0.29%, a 40–80 MB checkpoint. The orchestra barely got a booklet; the booklet got a few staples.

One layer, seen from the side:

code
Copy
            x (k-dim input)
            │
     ┌──────┴────────────────┐
     │                       ▼
     │            ══════════════════
     │            FROZEN  W₀ (d×k)      ──► W₀·x   (the highway:
     │            ══════════════════          all the original power)
     │                       │
     │                       ▼
     │            γ · ( B · A · x )        (the service road:
     │                                  ┌──┐
     │                                  │A │ r×k  funnel DOWN → r dims
     │                                  └──┘
     │                                  ┌──┐
     │                                  │B │ d×r  funnel UP → d dims
     │                                  └──┘
     │                                    (B starts at ZERO:
     │                                     no traffic on the
     │                                     service road at step 0)
     └──────────────────┬────────────────┘
                        ▼
             h = W₀x + γ·(B·A·x)

   at export:   W′ = W₀ + γ·BA     ← roadworks absorbed into the highway;
                                     service road torn down (zero latency)

Read the funnel as a compression story: information is squeezed through an r-dimensional bottleneck and unsqueezed. Anything expressible only through more than r independent directions simply cannot be learned — the constraint is the regularizer. And at deploy time, the two funnels multiply out into one d×k delta that gets added to the frozen matrix — a one-time offline arithmetic step, invisible to inference.

Continuing the orchestra from topics 175–176: full fine-tuning retrained every musician; generic PEFT handed out an annotation booklet that the players must re-read at every concert (adapter modules and prefix tokens sit in the performance path — that's their inference tax).

LoRA's booklet is different in one decisive way: the publisher prints your annotations directly into the next edition of the score.

• The original printed score is untouched (W₀ frozen) — no copyright fight with the composer, no risk of ruining it.
• Your notes page is tiny — 16 pages of bowing corrections for a 2000-page symphony (rank 16 out of 4096 dimensions).
• The booklet starts empty (B = 0): first rehearsal is identical to the original score. Your notes accumulate gradually.
• When you're happy, the corrections are typeset into the reprint (W′ = W₀ + γBA). From then on it is just the score — conductors perform it with zero extra effort, and any hall (inference server) and any abridged paperback (quantizer) accept it unchanged.
• Keep the original plus the note page and you can also run different note pages for different gigs (multi-adapter serving), or combine two note pages (adapter merging).

Every remaining section is about this booklet: how many pages to order (r), how loud the notes read (α), which sections get notes (target modules), the printing glitches people fixed (rsLoRA/LoRA+/DoRA), and the reprint policy (merging and governance).

The four dials on every LoRA config, with current (2024–2026) practice:

• Rank r — how many directions of change you buy. 8–64 covers most chat/format tasks on 7B–13B models; r=16 is the community default. Scale with task complexity: code and reasoning-heavy adapters benefit from r=32–64. The well-known failure mode in 2022 papers: r too small to carry reasoning updates, and the bottleneck silently clips the capability you asked for.
• Alpha (α) — the gain on the branch; an effective learning-rate multiplier. Community convention: α = 2r (r=16, α=32 → γ=2). Changing α without retuning LR re-scales gradients into the adapter — it is not a free knob, it moves the same lever as LR.
• Target modules — where the funnels attach. The original "only the Q and V attention projections" heuristic is outdated. 2024–2026 practice targets all linear layers (attention Q/K/V/O plus MLP up/gate/down): MLP coverage matters disproportionately for injecting knowledge and reasoning style. PEFT exposes target_modules="all-linear" for exactly this.
• Dropout — 0.05 on adapter inputs helps small datasets; often 0 for large instruction sets.

The code block is the canonical 2025 recipe for a 7–8B instruct model.

One caution the variants exist to fix: with the standard γ = α/r scaling, adapter updates progressively weaken as r grows (each of more directions gets proportionally less gain). If you raise r and see nothing happen, that's often why — see the callout and section 7.

from peft import LoraConfig, get_peft_model

cfg = LoraConfig(
    r=16,
    lora_alpha=32,            # gamma = alpha/r = 2
    target_modules="all-linear",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(base, cfg)
# 7B model => ~0.3% trainable, checkpoints ~40-80 MB

First, dynamics: LoRA tolerates learning rates ~10x higher than full fine-tuning — 1e-4 to 4e-4 where FFT used ~1e-5 — because (a) the frozen base cannot be destroyed by a big step, and (b) the branch starts at zero, so aggressive motion only reshapes a small residual. Catastrophic forgetting (topic 175's nightmare) becomes structurally limited: general capability was never in the blast radius.

Then the ecosystem. The 2023–2026 variants each fix one pressure point of the core equation:

• QLoRA (arXiv:2305.14314): LoRA on top of an NF4-quantized frozen base — the base itself shrinks ~4x, so one 24–48 GB GPU fine-tunes 33B–65B. Full topic 178.
• DoRA (arXiv:2406.06162): decomposes ΔW into magnitude and direction updates (learning "how strongly" separately from "which way"), consistently beating LoRA at low ranks.
• rsLoRA (arXiv:2312.03732): the rank-stabilized fix — scale by α/√r instead of α/r so high-rank training keeps signal strength.
• ReLoRA (arXiv:2307.05691): periodically merge, restart a fresh adapter, and keep going — cycling merge-and-restart so very high ranks train stably, aimed at full-FT-parity regimes.
• AdaLoRA: spends the rank budget where it pays — allocating r per matrix by singular-value importance and pruning uninformative directions during training.
• S-LoRA / multi-LoRA serving: batch thousands of distinct adapters against one frozen base in a single forward pass — the mechanism behind hosted fine-tune platforms; vLLM and SGLang ship native multi-LoRA request routing.

Because adapters are additive in weight space, they compose like numbers:

• Merge two trained adapters into one model — e.g., a code adapter + a JSON-format adapter (both are ΔW's; add them). Naive sums can conflict — the deltas interfere — which is the research area of task-vector arithmetic and merging rules like TIES (trim signs, elect merge) and DARE (drop-and-rescale sparse deltas).
• Merge into the base, then quantize the result: W′ = W₀ + γBA, then ship GGUF/AWQ/etc. The reprint step means every existing inference stack and quantizer accepts a LoRA-tuned model as just-a-model — the differentiator versus adapters/prefix tuning.
• Unmerged, the branch stays live at inference: correct for hot-swapping tenants, but it adds per-layer branch compute.

Operationally, LoRA gives unusually clean governance for ML engineering:

• The base model is versioned once (the printed score), signed and security-reviewed.
• Each adapter is an auditable, deletable artifact — 40 MB, diff-able, testable in isolation.
• Rollback is unloading one small file: a fine-tune incident (data poisoning, capability regression) doesn't require re-exporting or retraining a 70B model. Caveat from topic 176: after you merge, the delta is baked into the new edition — roll-back-ability trades against the latency win, so decide per deployment.

Final honesty note (from topic 176's ceiling): LoRA never corrects the base — factual errors and deep reasoning limits stay under the adapter. When the score itself is wrong, no booklet fixes it; that is when you pay for topic 175.