Word2Vec I: CBOW and Skip-gram Architectures

Last concept (word embeddings — in one plain sentence: give every word a list of numbers where near numbers mean near meanings) ended with a magic trick nobody explained:

Nobody hand-wrote them. Someone had to learn them — from raw text, with no dictionary of meanings, no labels, no supervision at all.

And there is a nasty constraint: you cannot train "know the meaning of cat" directly. Meanings are not measurable quantities. A training signal has to be something checkable, automatic, and everywhere in text.

Word2Vec's answer is one of the cleverest moves in machine learning:

The game? Guess the missing word.

This concept builds that game — both versions of it — plus the engine that made it trainable on billions of words.

Word2Vec (Mikolov et al., 2013, arXiv:1301.3781) is deliberately tiny. Forget everything you have heard about "deep" networks:

It is shallow on purpose: the network itself is worthless — nobody deploys it. It exists only to pressure the weights into carrying meaning, and then we throw the network away and keep the weights (rows of the input matrix are the map coordinates).

Why no nonlinearity (for CBOW)? Because a deep net would let the model learn a procedure for the game — and then the procedure, not the weights, would carry the knowledge. A shallow net cannot cheat: the only way to get good at peek-a-boo is to store statistics in the vectors themselves. Boring architecture, honest learning.

Training walks a sliding context window over raw text. For the sentence

the quick brown fox jumped over the lazy dog

with window 5, the center "fox" is paired with context slots {quick, brown, jumped, over}. Move the window; do this for every position of every sentence. That is the entire dataset — no parse trees, no dictionaries, no labels.

Why should this game produce a meaning-map? Because getting good at it forces the distributional hypothesis into the weights: if "fox" and "wolf" appear near the same words (chase, forest, fur), the same correct guesses pull their vectors toward the same neighborhood. Word2Vec is Firth's idea made optimizable (Concept 130, again in one sentence: similar company → similar meaning).

Same window, same tiny network, opposite questions.

CBOW (Continuous Bag of Words): show the neighbors, hide the center.

code
Copy
  the  quick  [ ??? ]  jumped  over        ← context given
  average the context embeddings → predict:  fox

Average the context-window embeddings, pass through one dense layer, softmax over the vocabulary, predict the center word. One prediction per window; mathematically a bagging of context (order inside the window is ignored).

Skip-gram: show the center, ask for the neighbors.

code
Copy
  [ fox ]  →  predict: the? quick? brown? jumped? over?
  each context slot = one separate prediction task

Take the center embedding, softmax, and predict each word in the surrounding window independently. d context slots = d separate loss terms.

The callout version to memorize:

Imagine a party where you are secretly profiling guests. You never see name tags with personality — only who stands next to whom.

Guest "fox" keeps being spotted with quick, brown, jumped, over. Another guest "wolf" is spotted with almost the same crowd: quick, grey, prowled, over.

CBOW round. You see the group {quick, brown, jumped, over} and must name the missing guest. Your current profile guesses "fox" — correct. You nudge fox's profile a little toward the crowd's average, and shove the wrong candidates away.

Skip-gram round. You see the single guest "fox" and must predict each companion separately: quick (yes), brown (yes), and a random stranger "calculator" (no). Each yes/no comparison nudges fox's profile toward its real companions and away from impostors.

Do 30 billion rounds of either game over the same corpus and a strange thing happens: guests that keep producing the same correct guesses end up with nearly identical profiles. Similarity of meaning has crystallized out of nothing but party-company statistics.

That is the whole trick of word2vec, and every embedding model since — including the contrastive sentence encoders powering RAG today — is a variation on "pick an auto-scored game, keep the by-product."

Both models share the two-matrix bookkeeping:

• W: rows = "input" vectors (what gets looked up when a word acts as source/center).
• W*: rows = "output/context" vectors (what a word's vector as a target is).

After training, rows of W are the published embeddings; W* is discarded (though concatenating or summing both sometimes performs better — an early "lesson learned" of the embedding era).

Empirical behavior from the original papers (and replicated countless times since):

• Speed: CBOW trains substantially faster — fewer updates: one center prediction per occurrence vs. roughly 2x window (≈8-10) skip-gram predictions per word.
• Frequency sensitivity: CBOW's averaging acts like smoothing — better for frequent words. Skip-gram's repeated, less-noisy updates are better for rare words, small corpora, and large datasets overall (the paper's headline: skip-gram wins on rare-word analogies).
• The 2013 follow-up "Efficient Estimation of Word Representations" plus Goldberg & Levy's 2014 review (ACL-2014, "word2vec Explained") established the practical defaults: window 5, skip-gram, 100–300 dimensions — which is why the famous GoogleNews-vectors-negative300.bin (2013) is a skip-gram, 300-d model.

In one table:

code
Copy
                 CBOW                 SKIP-GRAM
question     neighbors → center     center → neighbors
updates/word      1                    ~8-10
loves            frequent words        rare words
speed            faster                slower
analogy quality  decent                better (esp. rare)

Both models output a softmax over all V words at every prediction. With V ≈ 10⁵ and billions of training sentences, a naive softmax costs a full V x d matmul plus a V-way partition-function gradient per sample. For the 1.6B-word Google News corpus that meant weeks of compute.

Fix 1 — Hierarchical softmax (2013 paper #1): replace the V-way softmax with a path through a binary Huffman tree over the vocabulary (frequent words near the root). To score a candidate, you take a few yes/no branch decisions instead of comparing against all V words. Cost drops from O(V) to O(log V) per prediction — but gradient signals for rare words became weak (they live deep in the tree, few updates reach them meaningfully).

Numbers make it vivid: with V = 100,000, one softmax = 100,000 comparisons per step. The tree path is about log₂(100,000) ≈ 17 yes/no decisions. Same answer, four thousand times cheaper — except rare words sit 40+ levels deep and starve for training signal.

Fix 2 — Negative sampling (the 1310.4546 paper; Rong 2014 gave the noise-contrastive-analysis derivation): never compute the full softmax at all.

For each (center, context) positive pair, sample k ≈ 5–20 random "negative" words — drawn from a frequency^0.75 unigram distribution — and train a logistic classifier to score the pair as real vs. corrupted:

loss = −log σ(u·v) − Σⱼ log σ(−u·vⱼ)

In plain words: push this companion's score up; push k random impostors' scores down. Nothing else gets touched.

The party analogy says why this works: you do not profile a guest by ranking them against all 100,000 attendees. You show them next to their real companion and ten ringers, and just learn to tell real pairs from fake ones.

This is a huge, elegant trick: each update touches only k+1 rows of both matrices instead of V, so training scales to billions of words on a laptop-class multi-core machine — fastText and Gensim trained on 500B+ Common Crawl tokens with it.

A modern reproduction (Gensim) exposes every lever the papers studied:

• Subsampling frequent words (sample=1e-4): "the", "of" would dominate every window; skip most of their occurrences probabilistically — speeds training ~2-3x and boosts rare-word quality. (At the party: you stop re-testing profiles on the guest who stands next to literally everyone.)
• Min-count (~5): prune hapax legomena (words seen fewer than 5 times) from the vocabulary before training — no reliable profile can exist for someone you met once.
• Negative distribution: unigram^0.75, the word2vec paper's empirical choice that balances frequency against surprise — popular impostors are harder negatives, but ultra-rare ones still get some shots.
• Window: 5 for general semantics; larger windows drift toward topical similarity (words that share topics rather than grammar).

Word2Vec still earns its place in 2024–2026 stacks as cheap, interpretable lexical vectors: trained in minutes on domain text, used for misspelling/alias resolution, query-expansion in search, entity linking features, and as cold-start item embeddings.

It lost the representation-quality crown to contextual models (Concept 133) — but its objective, local window prediction + contrastive negative sampling, is literally the template modern contrastive embedding models still use. If you train a sentence-embedding model for a RAG stack today, you are playing a grown-up version of the party game in section 4.

from gensim.models import Word2Vec

model = Word2Vec(
    sentences=sentences,      # iterable of tokenized lists
    vector_size=300,          # embedding dimension d
    window=5,                 # context radius
    min_count=5,              # prune hapax legomena
    sample=1e-4,              # frequent-word subsampling
    sg=1,                     # 1 = skip-gram, 0 = CBOW
    negative=10,              # negatives per positive pair
    epochs=10,
    workers=8,
)
print(model.wv.most_similar("server", topn=3))