Vocabulary Size: Coverage vs. Compute in Tokenizer Design

You have now met the vending machine (tokenization) and its menu-writing algorithm (BPE, WordPiece, Unigram).

Every menu has a length. The tokenizer for BERT carries 30,522 pieces. GPT-4o carries roughly 200,000. Gemini carries about 256,000.

How was that number chosen? And why should you care?

Pick V too small and everything gets chopped into too many pieces — Japanese text can cost 3–6x more tokens than it should, and every one of them is billed.

Pick V too big and you pay a fixed tax in weights and compute on every single step — plus a long tail of vocabulary rows that barely trained at all.

This concept is about that trade. It is a two-sided lever, and once trained, the lever is welded in place.

First, where does V actually live inside a Transformer? Almost nowhere — exactly two places, both at the edges of the network:

1. Input embedding matrix: shape V x d. V rows, each a d-dimensional vector. (From the embeddings concept: a token's ID just picks its row.)
2. Output projection ("unembedding"): another V x d matrix, unless tied with the input. Its job is to score every vocabulary item at every step so the softmax — the layer that turns scores into probabilities — can pick the next token.

Everything in the middle (attention heads, MLPs, layers) does not care about V at all.

So the mental model:

Front door: numbers → vectors. Back door: vectors → probabilities over all V candidates.

Let's put real numbers on the doors.

Part A — the parameter bill.

LLaMA-3 has V = 128,256 and hidden size d = 4,096.

V x d = 128,256 x 4,096 ≈ 525,000,000

That is ~0.53 billion parameters in the embedding matrix alone. LLaMA-3 unties its output projection, so the back door is another 0.53B.

More than an eighth of the entire model is the menu.

Part B — the token bill.

Take the word internationalization.

• Under a small 30k English-leaning vocabulary it might shatter into inter + na + tion + al + ization → 5 tickets.
• Under a 200k multilingual vocabulary it is plausibly 1 ticket.

Same meaning. Five times the positions. And positions are where everything hurts: attention cost grows quadratically with sequence length, context windows fill up 5x faster, and per-token API pricing multiplies by five.

Same lever, opposite ends: Part A is the fixed cost of a big menu, Part B is the variable cost of a small one.

# LLaMA-3-8B: 128,256 vocab x 4096 hidden, untied embeddings
V, d = 128_256, 4096
print(f"Embedding table:  {V*d/1e9:.2f}B params")   # 0.53B
print(f"Unembedding:      {V*d/1e9:.2f}B params")   # 0.53B (untied)
# vs bert-base-uncased: 30,522 x 768 = 0.023B — tiny because encoders
# classify with small heads instead of predicting over the full vocab

The whole topic balances on one seesaw:

code
Copy
        SMALL V                          BIG V
   ┌────────────────┐              ┌────────────────┐
   │ tiny doors     │              │ 0.5B+ params   │ ← fixed tax,
   │ cheap softmax  │              │ fat logit layer│   every pass
   │                │              │                │
   │ every rare word│              │ common words   │
   │ shatters into  │  ◄──seesaw──►│ fit in ONE     │
   │ many tokens    │              │ token          │
   │ (3-6x for      │              │                │
   │  Japanese!)    │              │ long tail of   │
   └────────────────┘              │ cold rows      │
                                   └────────────────┘

And the industry has been visibly sliding right along that lever:

code
Copy
2018    BERT          30k
2019    GPT-2         50k
2023    GPT-3.5/4     100k
2024    GPT-4o        200k     LLaMA-3  128k
2024+   Qwen          152k     Gemini  ~256k
        ────────────────────────────────────────►
        vocab sizes grew ~5–10x in six years,
        driven by multilingual + code economics

Nobody did this out of greed for big numbers. Each jump paid for itself in shorter sequences.

Carry one picture through the rest of this concept: a restaurant kitchen deciding how many dishes to pre-cook.

The vocabulary is the set of pre-made dishes. A sentence is an order the kitchen assembles by platter-combining.

• Many pre-made dishes (big V): most orders need only two or three platter grabs — service is fast, the counter (context window) empties quickly. But the rent is brutal: every dish needs its own cabinet slot (an embedding row), and the waiter must check every dish before recommending one (softmax over V).
• Few pre-made dishes (small V): rent is cheap. But every order of "internationalization" requires scrambling five ingredients together from basic stock — slower service, longer lines, and the same popular dish arrives at the table in fragments.
• Rare dishes (the long tail): a dish ordered 5 times all season never gets practiced. Its cook (the embedding row) learns almost nothing from those 5 orders — it stays close to raw ingredients, i.e., a near-random vector the model then misuses.

Three bills, one kitchen. Keep them separate:

Growing V cuts prep time, raises rent, and dilutes practice.

A vocabulary too small for its training distribution fragments text: common words get split into several pieces, so every sentence costs more tokens. The measured numbers:

• English under a 30k WordPiece vocabulary averages ~1.4–1.6 tokens per word; under a modern 128k–200k byte-level BPE, ~1.2–1.3.
• For morphologically rich or non-Latin languages the gap explodes: Japanese or Chinese under an English-heavy 50k vocabulary can run 3–6x more tokens than under a 128k+ multilingual vocabulary.
• Longer sequences mean: quadratic attention cost, faster context-window exhaustion, slower inference, and higher price per unit of meaning. This is why tokenizer choice was the headline complaint about early English-centric models serving global traffic.

The trend line in section 4 is this ledger in motion: BERT (2018) 30k → GPT-2 (2019) 50k → GPT-3.5/4 (2023) 100k → GPT-4o (2024) 200k, LLaMA-3 (2024) 128k, Qwen 152k, Gemini ~256k — vocabularies grew ~5–10x in six years, driven almost entirely by multilingual and code economics.

Compute per step also scales with V. The final projection produces a V-wide logit vector and the softmax cross-entropy sums over all of it. For large V and long contexts, frameworks shard this computation across GPUs (tensor/vocabulary parallelism, e.g., Megatron-LM) precisely because the V x d matmul is the single widest linear layer in the stack.

Why not just ship a 1M-token menu anyway? Four costs bite:

• Embedding-table cold start: each row is learned from that piece's occurrences. A token appearing 5 times in 15T training tokens gets a nearly-random vector. Huge vocabularies fill the long tail with undertrained entries that models then misuse or hallucinate IDs for — the "cold dishes" of the kitchen analogy.
• Softmax dilution and memory: logits are V floats per position. At V = 250k, the logit tensor for one 128k-token sequence in fp32 runs to hundreds of GB without sharding — a real serving constraint in 2024–2026 long-context inference.
• Diminishing returns: merge-based vocabularies follow Zipf logic. Past roughly 64k–128k for broad multilingual text, each additional merge covers vanishingly less corpus mass, while parameter cost keeps growing linearly.
• Tokenizer fairness: bigger vocabularies change tokenization statistics so aggressively that cross-model comparisons must switch to bytes-per-token (BPT) or bits-per-byte (BPB) rather than raw perplexity or tokens-per-word (see the Perplexity concept). Two models with different V are literally measuring "per position" over different amounts of text.

Rule of thumb to memorize:

How teams actually decide, in order:

1. Target languages + corpus mass. English-only, budget-constrained encoder: 30k–64k. Production multilingual chat model: 128k–256k. Code-heavy assistants push the top of that range, because identifiers like getOrNullAsync only merge into single tokens when V is big enough to afford them.
2. Parameter budget check. V x d must stay a sane share of total capacity — many labs cap the vocab at ~10–15% of parameters. Tied embeddings (one matrix serving both doors) halve the cost: GPT-2 ties; LLaMA-3 unties.
3. Measure coverage, not vibes. Compute tokens-per-character (or BPT) per script on held-out traffic. If Japanese costs 4x English on your candidate V, raise V or rebalance training-corpus sampling. Multilingual vocabularies allocate dedicated token mass to scripts — which is why Gemini can spend ~256k rows without hurting English.
4. Algorithm pairing. BPE/WordPiece stop at the target size; Unigram (SentencePiece) starts oversized and prunes by likelihood, giving smoother coverage curves at large V — one reason T5, ALBERT, LLaMA-1/2, and Gemma picked it.

And back to the kitchen for the closing lesson: the menu is printed before the restaurant opens, bolted to the wall forever. LLaMA's jump from 32k (LLaMA-2) to 128,256 (LLaMA-3) was not a settings tweak — it re-baked the whole model. Choose V the way you would choose a building: cheap to plan, ruinous to change.