Adapter Modules

Imagine you have one good pretrained model and fifty jobs for it.

• Sentiment for the legal team.
• Spam filtering for support.
• A German version, a Japanese version, a Hindi version…

Full fine-tuning means fifty copies of the whole model. For BERT-large that is fifty × ~340 million parameters. Fifty brains, each one slightly better at one job.

Something like: keep one frozen shared brain, and bolt on a tiny per-task "sticker" that steers it?

That is exactly what adapters did. Houlsby et al. (2019) inserted a bottleneck feed-forward network after every (or alternating) sub-layer of BERT — down-projection to d/r dimensions, ReLU, up-projection back to d, each adapter surrounded by its own residual connections so it begins as a transparent identity:

x → [Attention; Norm] → Adapter(x) → [FFN; Norm] → Adapter(x)

With reduction factor r ≈ 32 on BERT-large, adapters added 3.6% parameters yet matched full fine-tuning quality on GLUE, and beat it in low-data and multi-task regimes (the task deltas stayed isolated).

This was the first demonstration that transfer learning did not require moving billions of weights — the intellectual ancestor of LoRA.

So the question this topic answers is

An adapter is simply

Unpack the pieces:

• Down-projection shrinks the vector: d → d/r. If d = 3072 and r = 32, the adapter squeezes every activation down to 96 numbers. Squeezing forces the module to keep only what matters.
• Non-linearity (ReLU or GELU) in the middle — the same ingredient ordinary MLPs use to model non-straight relationships.
• Up-projection expands back: d/r → d, so the output has the original width and can be added into the stream.
• Residual connections on both sides: output = (sub-layer result) + Adapter(sub-layer result). The adapter adds a correction, it does not replace the signal.
• Zero-initialized up-projection: at step zero, the up-projection outputs all zeros, so Adapter(x) = 0 and the model is bit-identical to the frozen original. Learning starts from "do nothing" and grows from there.

One line to remember:

frozen main path + trainable detour, starting at zero.

Compare that phrasing to LoRA (you will meet it in one sentence here): LoRA also learns a low-rank correction that starts at zero — but it attaches the correction to the weight matrices themselves, while adapters attach theirs to the stream of activations. Same math spirit, different wall to hang the shelf on.

Take one transformer layer with width d = 3072 (BERT-large scale) and reduction factor r = 32.

Step 1 — bottleneck width:

d/r = 3072 / 32 = 96

Step 2 — the adapter has two matrices:

• down-projection: 3072 × 96 ≈ 295K parameters
• up-projection: 96 × 3072 ≈ 295K parameters

Total per adapter ≈ 590K parameters.

Step 3 — Houlsby places two adapters per layer (one after attention, one after FFN): ≈ 1.18M trainable per layer. The frozen layer itself holds tens of millions. Across the whole model the trainable slice lands at ~3.6% of full fine-tuning — that is the famous number.

Step 4 — why does r control everything? Try r = 4:

d/r = 3072 / 4 = 768 → each adapter ≈ 4.7M parameters — 8x bigger, more capacity, more memory. Smaller r (fewer squeezed-out dimensions) = bigger adapter. That is the capacity dial.

And the transparency check, with tiny numbers. Suppose an activation is x = [1.0, 0.5] and the up-projection is zero-initialized:

Adapter(x) = up(ReLU(down(x))) = 0·(something) = [0.0, 0.0]

output = x + [0, 0] = [1.0, 0.5] — unchanged. Training then nudges up away from zero, and the correction grows exactly as much as the task needs.

Draw one transformer block. The big boxes are frozen; the small ones are trainable.

Serial (Houlsby) — the data walks through an adapter after each sub-layer:

code
Copy
x ─► [ Attention/Norm ] ─►( + )─► [ FFN/Norm ] ─►( + )─► out
                            ▲adapter▲              ▲adapter▲
                     (trainable detour, in sequence)

Parallel (Pfeiffer-style) — the adapter runs alongside the whole block, and its output is summed once:

code
Copy
            ┌─ [ Attention/Norm ] ─┐
x ──────────┼─ [ FFN/Norm        ] ┼─► ( + ) ─► out
            └─ [ adapter branch  ] ┘   (one sum)

See the trade in one glance:

• Serial: two extra sequential modules per layer → more quality (the original paper's finding), but every token waits for four hops instead of two → inference latency.
• Parallel: one extra branch, computed alongside the frozen path → a single extra matrix path, serving-friendly latency, a minor quality trade.

The residual (+) in both pictures is the safety belt: even if the adapter outputs garbage mid-training, the frozen signal still gets through.

Carry this through: the pretrained model is a grand, protected heritage train station you are not allowed to renovate.

Fifty tenants (tasks, languages, customers) each want a service counter. Knocking down walls (full fine-tuning) is out of the question — and fifty renovations would bankrupt you.

So you build small kiosks:

• A kiosk is tiny (the bottleneck: a queue squeezes from 3072 lanes down to 96 desks and back).
• Kiosks start closed for business (zero-initialized up-projection): the station works exactly as before on day one.
• Serial placement = tenants must visit the kiosk in the corridor between the two main halls — everyone walks through it; maximum influence, but it adds walking time for every passenger (latency).
• Parallel placement = the kiosk sits alongside the main hall, and its answer is merged at the exit gate — one extra path, no queue in the corridor.
• Routing = a signboard at the door directs each visitor to the right tenant kiosk (per-request specialization inside one station).
• Fusion = combine several kiosks' advice into a single summary desk for newcomers (zero-shot transfer to unseen tasks).
• The one thing you can never do: remove the kiosks and engrave them into the walls. They are separate structures in the flow of passengers — a non-linear module acting on activations — whereas LoRA is literally an extra number etched onto existing signs, which can be re-painted in (merged). That single architectural fact drives most of today's method choice.

Two axes define an adapter architecture (the lineage from Section 1 in technical terms):

• Serial (Houlsby): one adapter after attention and one after FFN, each with its own residuals — the original, slightly stronger, but adds two sequential modules per layer (latency at inference).
• Parallel (Pfeiffer / Houlsby "parallel" variant): a single adapter branch running alongside both sub-layers of the block, summed with the residual — one extra matrix path, better serving latency, minor quality trade.
• Reduction factor r: controls capacity; d/r down then back up. Typical r=8-64; smaller r saves memory but throttles complex tasks. AdapterDrop-style schedules skip adapters in early layers (early layers already generalize; save the compute).
• Initialization/dropout: the up-projection is initialized small/zero and dropout (~0.1) regularizes the branch — the same transparency logic LoRA later adopted with B = 0.

And one structural consequence that is worth stating precisely:

Because adapters are activations-space modules (they live in the forward graph, not the weight matrices), they cannot be merged losslessly the way ΔW = BA can — that is the structural difference driving today's method choice.

Adapters' residual-branch structure — separate kiosks, remember — made them the natural unit for composition:

• Multi-task islands: one adapter per task on a shared frozen base (the Houlsby paper's original setting), then adapter fusion (summing stacked task adapters) for zero-shot transfer to unseen tasks.
• Language-task lattices: the MAD-X ecosystem trained small language adapters and task adapters on top of one multilingual base (XGLM/XLM-R), letting a new language + task pair compose two tiny modules — a blueprint for low-resource NMT and classification in 2021-2023 and still used for multilingual speech/LLM stacks. Picture it: one frozen multilingual brain + a grid of "language kiosk × task kiosk" combinations covering 55+ languages.
• Routing: learned or rule-based gating selects which adapter(s) handle each input, giving per-request specialization inside a single deployment.

In 2024-2026 this pattern survives in unified speech models (Whisper-style encoders with language/tooling adapters), encoder-only enterprise NLU stacks, and libraries like AdapterHub.ml / PEFT AdapterConfig.

The honest decision table:

Pick adapters over LoRA when

• (a) you want hot-swappable per-tenant branches and can tolerate the un-merged residual compute (batched multi-adapter inference engines handle this);
• (b) you need explicit composition of independently trained capability modules (language × task lattices, fused skills);
• (c) your stack (encoder models, seq2seq, speech) has adapter tooling maturity.

Pick LoRA/QLoRA when

• it is a decoder-only LLM chat/format fine-tune with strict latency SLOs — mergeable weight deltas mean zero inference overhead after merging, which is the structural reason LoRA dominates there.

A pragmatic 2025 recipe even trains both: adapter + LoRA hybrids (e.g., LoRA on projections, Houlsby adapters in FFN blocks) squeeze the last points before full fine-tuning.

The adapters did. LoRA inherited the zero-start correction idea, moved it into weight space, and won the latency war — but every "one base model, many small modules" serving design you see today is walking around in kiosk form.