QLoRA: Quantized Low-Rank Adaptation

You run a small AI shop. A customer wants a 65-billion-parameter open model (think Llama-1 65B) fine-tuned on their data.

You already know one trick: LoRA. In one plain sentence: instead of updating all the weights of the model, LoRA freezes the model and trains a tiny extra pair of matrices on top, so only a fraction of a percent of parameters ever receive updates.

But here is the catch nobody warns you about.

LoRA shrinks the trainable slice. It does not shrink the frozen base.

That giant base model still sits in GPU memory in BF16 — 2 bytes per parameter:

• 65B parameters × 2 bytes = 130 GB of weights alone.
• Add gradients, AdamW optimizer state, and activations, and a full fine-tune needs roughly 1.1 TB.

Your machine? One 48 GB RTX A6000. Or one 80 GB A100.

So the question becomes

Dettmers et al. (2023) asked exactly this, and QLoRA is the answer: keep the frozen base at 4 bits per weight.

The headline result: training a 65B model dropped from ~1.1 TB (full fine-tuning) to a single 48 GB GPU. Today the same recipe family (via bitsandbytes, Unsloth, torchao) fine-tunes 70B models on 48 GB cards and 7-8B models on 12-16 GB consumer GPUs. No server cluster. No new hardware. Just a smarter way to store the numbers.

QLoRA is simply

Three pieces. Let's unpack them one by one.

• Piece 1 — storage: 4 bits. The base model's weights are quantized once, offline. Quantizing means: replace each precise number with the nearest value from a small menu of allowed values. NF4's menu has 16 entries (4 bits = 2⁴). Half a byte per weight instead of two → the base takes ~4x less memory.
• Piece 2 — compute: still 16 bits. A 4-bit weight is useless for math, so right before the GPU multiplies, it dequantizes a small block of 64 weights back to BF16 on the fly. Storage is tiny; the arithmetic stays full quality. Use one item, fold it back up.
• Piece 3 — learning: only the adapter. The quantized base is frozen — gradients never touch it. All gradient updates land on the BF16 LoRA branch. Because the trainable slice is ~0.1-1% of parameters, the AdamW optimizer state (which is normally the biggest memory hog) shrinks to almost nothing.

Read it as one pipeline:

store in NF4 → unpack block to BF16 → multiply → send gradients → only LoRA changes.

Notice what QLoRA did not change: the LoRA math from the LoRA topic is exactly the same. QLoRA attacked the other half of the memory bill — the frozen base — with a better number format.

Forget 65B for a second. Take a tiny model with 8 weights, one block.

Suppose the weights (already scaled so the biggest magnitude is 1.0) are:

[0.90, -0.45, 0.15, -0.30, 1.00, -0.10, 0.25, -0.60]

BF16 storage: each weight gets 2 bytes → 8 × 2 = 16 bytes.

NF4 storage: each weight gets 4 bits = 0.5 bytes → 8 × 0.5 = 4 bytes. Each weight is snapped to the nearest of 16 allowed levels, plus the block stores one scale factor (its largest magnitude, 1.00 here, in BF16). The rounded values are close — NF4 was built so that "close" is as good as it can get for typical weights.

Now scale the same math to a 65B model:

• BF16 base: 65,000,000,000 × 2 B ≈ 130 GB
• NF4 base: 65,000,000,000 × 0.5 B ≈ 32.5 GB

That one move — 130 GB down to about 35 GB — is most of why QLoRA fits on a single 48 GB card.

Yes! One BF16 scalar per 64-weight block. That overhead matters at 65B scale, which is exactly why QLoRA adds a second trick: quantize the scale factors themselves (double quantization — Section 6).

And the gradients? A quick sanity number: if LoRA rank 64 adapters cover ~0.3% of a 65B model, the trainable + AdamW footprint is a few hundred million bytes per million weights — under ~1-2 GB. The base (35 GB) plus tiny adapter (<1 GB) fits on one 48 GB GPU. Full fine-tuning instead pays ~1.1 TB, because every one of 65B parameters would need gradients and two AdamW states in high precision.

Most people picture training as "everything lives at full precision." QLoRA splits the picture in two:

code
Copy
MEMORY (storage)          COMPUTE (math)           LEARNING (updates)
──────────────────        ──────────────────       ───────────────────
base: NF4 4-bit   ──►     unpack block of 64  ─►   BF16 matmuls
(frozen, read-only)       to BF16 on the fly       │
                                                   ▼
adapter: BF16     ◄────── gradients flow here ◄───┘
(LoRA branch)            — the base never updates

And the memory bars themselves tell the business story:

code
Copy
Full BF16 FT (65B):  ███████████████████████████████████  ~1.1 TB  ✗ no GPU fits it
QLoRA (65B):         ███ ~35 GB base + <1 GB trainable    ✓ one 48 GB GPU

The picture to keep in your head:

• The wide bars are weights being read — enormous, but frozen and compressed.
• The thin arrow is everything the optimizer touches — tiny, because only LoRA learns.

That asymmetry — huge read-only base, tiny writable adapter — is the whole reason 4-bit storage is safe here. Mistakes in the frozen base are frozen too (see the Section 1 callout): the adapter just learns around them.

Carry one analogy through the rest of this topic: moving into a tiny apartment with all your furniture.

• Your sofa is a 65B-parameter model. The apartment is a 48 GB GPU. The sofa simply does not fit.
• So you put the sofa in a vacuum bag and suck out 75% of the air. That is NF4 quantization: same sofa, 4x less space. You did not throw anything away; you compressed it.
• When you actually want to sit on it, you unpack one piece at a time, use it at full comfort (BF16 math), then fold it back. Nobody sits on a vacuum-sealed sofa — that is the dequantize-block-on-the-fly step.
• The things you do change in the apartment — cushions, throws, a new lamp — are the LoRA adapter. Small, cheap to swap, and the only items the moving budget (AdamW state) has to track.
• Each vacuum bag carries a printed label: "this sofa = 1.0 units wide." That label is the block scale factor. Double quantization = you have thousands of labels, so you compress the label list itself into a single index card.
• On packing day the boxes briefly overflow the apartment. Paged optimizers = you own a storage unit down the street (CPU RAM): overflow goes there for the spike, then comes back. Nothing gets thrown into the dumpster (no OOM crash).

Everything technical below is just this story with numbers attached.

NF4: a 4-bit format matched to weight statistics.

Trained-network weights are approximately normally distributed with ~zero mean — a bell curve centered on 0. A standard INT4 uniform grid wastes code points in the sparse tails and crowds the dense center. NormalFloat4 (NF4) is a 4-bit data type whose 16 quantization levels are the information-theoretically optimal quantiles of a normal distribution (derived from the expected quantiles, 75th-percentile normalization), making it unbiased for Gaussian block data.

Plain words: uniform INT4 asks "which 16 evenly-spaced numbers do I allow?" NF4 asks "at which 16 values would a bell-curve crowd be best represented?" Same 4 bits, better-placed levels, less error.

QLoRA also applies quantization at the right granularity: block-wise — each block of 64 weights gets its own absmax-derived scaling factor stored in BF16, so outliers in one block do not compress the resolution of another. During the forward/backward pass, blocks are dequantized to BF16 on the fly inside custom kernels: storage is 4-bit, compute stays 16-bit.

Double quantization (the label-index-card trick). The per-block BF16 scalars add ~0.5 bytes/param overhead (0.125 B from 4-bit data + 0.5 B from 64-block scaling). Quantizing those scalars themselves into 8-bit FP8 blocks amortizes the constant to ~0.13 bytes/param — netting ~0.37 bytes saved per parameter, ~1/3 of the total footprint. For 65B that is ~24 GB recovered.

Paged optimizers (the storage unit). Optimizer steps occasionally spike memory (e.g., gradient accumulation boundaries). QLoRA moves AdamW states into pinned CPU RAM via unified memory when pressure rises, converting catastrophic OOM crashes into slower-but-successful steps.

Batched 4-bit matmuls. Custom tiny CUDA kernels dequantize + multiply per block so throughput stays practical despite the storage/compute split.

The whole stack is a few lines of config, because the community tools (bitsandbytes + PEFT) do the heavy lifting:

QLoRA is the standard "one GPU, big model" recipe, but it is not free. Two prices to know.

Price 1: speed. Every matmul pays for unpacking. Dequantization during compute makes steps ~2-4x slower than BF16 LoRA on the same hardware. So the honest decision rule is simple:

If yes → use plain LoRA; it is faster. If no → QLoRA turns "impossible" into "possible, just slower."

Price 2: deployment friction. Deploying a QLoRA result usually means merging the adapter into the base. Merging into the 4-bit base is impossible directly — a 4-bit matrix cannot absorb BF16 deltas; there is no room between the 16 allowed levels. In practice you merge onto a dequantized BF16 copy (quality close to trained) or export with GGUF/AWQ pipelines for inference.

Quality, though? The paper found rank-64 QLoRA essentially matched full BF16 fine-tuning on the instruction tasks it evaluated — a finding that reshaped open-source fine-tuning economics. Once the result is merged to BF16, serving pays nothing extra: the adapter is folded into ordinary weights, and the trained model can be quantized later with GPTQ/AWQ for deployment (fine-tune → merge → quantize → serve, in that order).

So QLoRA's place in 2024-2026 stacks is clear: it is the training-time trick — the cheapest way to teach a very old, very large dog new tricks on one card.