Prompt Tuning and Prefix Tuning

Picture the situation.

Your company pays for one hosted model. A hundred customers each want it to behave slightly differently — one for legal summaries, one for medical triage, one for playful marketing.

Fine-tuning (updating the model's weights on each customer's data) gives you a hundred copies of a giant model. Expensive. Slow. And sometimes flat-out impossible:

• The base weights are quantized (compressed to 4 bits, no room to edit).
• The base is shared by everyone on one server.
• The base is license-locked — you may use it, but not modify it.

So the question becomes

Yes. And the trick is surprisingly small: feed the model extra numbers that are shaped like its own internal representations, and train only those numbers.

These learned extra numbers go by names like soft prompts, virtual tokens, or continuous prefixes. Not words you typed — vectors of decimals, tuned by gradient descent, sitting where real word-embeddings would sit.

Two methods own this space:

• Prompt tuning — the learned vectors go in at the entrance (input embeddings only).
• Prefix tuning — the learned vectors go into the attention of every layer.

Weights untouched in both cases. That is the whole family's contract.

A frozen language model turns your words into embeddings: lists of numbers (vectors) that capture meaning. Prompt/prefix tuning simply adds extra, learnable vectors into that number-stream.

Unpack each in one line:

• Prompt tuning (Lester et al., 2021) freezes the entire model and trains only P soft token embeddings prepended to the real input, as if the sentence began with P words that exist in no dictionary.
• Prefix tuning (Li & Liang, 2019-2021) attaches learned matrices Kᵢ, Vᵢ per layer to the attention key/value states, as if the sequence began with Lᵢ invisible conditioning tokens whose behavior is decoupled from real tokens (their outputs are truncated).

A one-sentence refresher on the terms: in attention, each token publishes a Key (what I am) and a Value (what I offer); every other token scans the keys and mixes the values it cares about. Prefix tuning slips extra keys/values into that pool at every layer.

The family's motto:

task knowledge lives in continuous vectors, not in weights.

Say your model maps each token to an embedding with 4 numbers (real models use 768 to 12288, but 4 is easy to follow).

Real input: "This movie was great" → 4 tokens, 4 vectors each:

code
Copy
"This"   → [ 0.12, -0.90,  0.33,  0.01]
"movie"  → [-0.44,  0.21,  0.08,  0.77]
"was"    → [ 0.05, -0.02,  0.61, -0.19]
"great"  → [ 0.90,  0.14, -0.27,  0.42]

Now prompt-tune a soft prompt with P = 2 virtual tokens:

code
Copy
⟨s1⟩ → [ 0.63, -1.10,  0.04,  0.28]   ← a "word" no tokenizer can produce
⟨s2⟩ → [-0.35,  0.99,  0.71, -0.06]   ← these 8 numbers ARE the trainable task

Training sends gradients only into these 8 numbers. The model's billions of weights never move. At inference, you feed [⟨s1⟩, ⟨s2⟩, "This", "movie", "was", "great"] and the frozen model reads the two fake tokens first — they steer it into "sentiment mode".

Scale check, plain math:

• BERT-base embedding width = 768. A P = 20 prompt = 20 × 768 = 15,360 numbers ≈ 60 KB in FP32. That is your entire per-task artifact.
• At inference, those P fake tokens add P slots of context that attention must process. For P = 20 on a 512-token window, that is ~4% extra attention work. Tiny here — but see Section 7, because prefix tuning pays this at every layer.

The mental image: prompt tuning is learned words, prefix tuning is learned attention context at every floor of the building.

A transformer is a tall building: embeddings go in at the bottom, then pass up through layer after layer.

Prompt tuning = a sign at the entrance only:

code
Copy
   ┌─────────────────┐  layer N    (frozen)
   │      ...        │
   ├─────────────────┤  layer 2    (frozen)
   ├─────────────────┤  layer 1    (frozen)
   ├─────────────────┤  input
   │ [P][P][ words ] │  ← soft prompt P sits HERE, only here
   └─────────────────┘
   The task signal must survive the climb through every frozen layer.

Prefix tuning = a PA system on every floor:

code
Copy
   ┌─────────────────┐  layer N   frozen block + prefix KV #N
   ├─────────────────┤  layer 3   frozen block + prefix KV #3
   ├─────────────────┤  layer 2   frozen block + prefix KV #2
   ├─────────────────┤  layer 1   frozen block + prefix KV #1
   ├─────────────────┤  input embeddings (untouched, real words)
   └─────────────────┘
   Each attention layer gets a fresh injection of learned keys/values.

Read the consequences straight off the picture:

• Prompt tuning has one entry point → the conditioning is weakest (it only flows deeper via attention), but serving is free — the model runs unmodified and the prefix is a KB-sized tensor. Swapping tasks = swapping a tensor.
• Prefix tuning re-conditions every layer → more expressive, historically stronger on generation. The price: those prefix K/V states ride along in every attention computation at inference.
• One more quirk you can see in the diagram: prefix K/V outputs are discarded (the invisible tokens get truncated from the output). They are there to be looked at, never to speak.

The frozen model is a world-class chef who refuses cooking lessons — perfect skills, zero flexibility.

You want 100 different customers' dietary styles served by the same chef. So:

• Prompt tuning = you tape a note on the pantry door: "Table d'hôte for the guest in seat 7 — think dessert." One hint at the entrance. A great chef carries that thought through the whole menu; a mediocre one forgets it by the third course. This is Lester's scale finding in kitchen form: the bigger and more skilled the model, the better a single input hint works.
• Prefix tuning = a whisperer on every station — someone leans into each line cook's ear as they plate ("the guest is gluten-free", "the guest loves heat"). More reliable, more intrusive: the whispering costs every station a few extra seconds (per-layer prefix K/V in the attention path at serve time).
• Virtual tokens = notes written in a language with invented words. They are not in any recipe book (vocabulary), but the kitchen understands them perfectly because they are written in the kitchen's own handwriting (embedding space).
• Hard prompts / AutoPrompt = using only real dictionary words on the note — easier for humans to read (interpretable) but a much weaker, coarser tool: you cannot write "0.63" on a real word.
• The generator FFₚ (see Section 6) = you do not hand the whisperers a fixed script; you give them a one-page guideline and each improvises the exact words per floor. More stable than memorizing a full script line by line.

Two landmark findings defined prompt tuning:

• Scale closes the gap: with model sizes ≥ ~7B (T5-11B+ and later LLMs), a prompt vector initialized from repeated words (e.g., "?" tokens) matches full fine-tuning on Super-Natural Instructions while learning only 0.01-0.1% of parameters.
• Small models underfit: on BERT-base, prompt tuning trails fine-tuning notably — the conditioning signal is too weak to reorganize shallow networks without touching weights.

"Conditioning beats surgery at scale" is the headline of the whole method family.

Prefix tuning's stability design is the other must-know. To avoid instability, the prefixes are not stored directly — a small feed-forward generator FFₚ maps a shorter trainable tensor to the full prefix, and all gradients flow through FFₚ into the frozen model. Why? Directly optimizing full prefix matrices makes the loss landscape rough; the low-rank generator regularizes it (the same "train a small thing, let it produce a big thing" trick LoRA uses).

And because every layer is re-conditioned rather than the input alone, prefix tuning is more expressive and historically outperformed prompt tuning on generation tasks at medium scale. The cost: a per-layer cache of prefix KV states that rides along in every attention computation — measurable inference overhead that LoRA's mergeability does not suffer.

The family branches in useful directions:

• P-Tuning v1/v2 optimizes continuous prompts at input and deep layers (essentially prefix tuning dressed for GPT-style models), recovering much of the FT gap on few-entity NER and GLUE.
• AutoPrompt / gradient-guided discrete search proposes real vocabulary tokens by mask-model likelihood — interpretable but brittle. When you can read the learned prompt as English words, you get an audit trail; gradient search over a discrete vocabulary is inherently weaker than over continuous space, though.
• Soft prompts for frozen encoders: the 2022-2026 survival niche. Text-to-image ecosystems (e.g., tuning a frozen T5/CLIP text encoder with a small set of learned embeddings), vision-language connectors, and safety "system-prompt vectors" use prompt-style conditioning because the conditioned module is unmodifiable by design — quantized, shared, or license-encumbered encoders where adding modules to weights is impossible or unwanted.

That last bullet is the honest answer to "why learn these when LoRA exists?": prompt/prefix conditioning is the only PEFT family that works when the weights cannot be changed at all — you cannot merge anything into a 4-bit matrix, but you can always hand it extra input vectors.

Benchmark consensus across GLUE/seq2seq and modern instruction suites:

full FT ≥ LoRA ≥ adapters > prefix ≈ prompt tuning on quality, while the memory ranking inverts.

Prefix/prompt methods also carry inference-time overhead (extra tokens per layer) and composition quirks — soft prompts do not stack as cleanly as task vectors.

Still, they remain the go-to when

• (a) the model must stay bit-identical (regulated/frozen/quantized bases);
• (b) you need tiny per-tenant payloads (KBs, not MBs — a soft prompt is a tensor you can hot-swap per request);
• (c) you want a "learned system prompt" that is auditable as input-space conditioning.

In production LLM platforms they appear mostly as prompt-vector style safety/guidance modules and in multimodal glue layers rather than as the primary enterprise fine-tune — that role went to LoRA/QLoRA.

Decision ladder to remember:

The one-sentence lineage, useful in interviews: adapters taught us "tiny trainable modules work", prompt/prefix tuning taught us "even inputs can be the trainable module at scale", and LoRA then combined low-rank corrections with mergeability — which is why it won the default slot.