Contextualized Embeddings: Words Re-Read in Every Sentence
Static embeddings give every word one frozen address — "bank" the river and "bank" the loan live at the same coordinates. ELMo and then BERT fixed this by recomputing each token's vector from the whole sentence, trading a lookup for a forward pass and launching the entire modern NLP stack.
01.The Problem: One Address, Two Completely Different Places
We finished the last three concepts (embeddings, word2vec, GloVe — in one plain sentence each: words get vector "addresses" learned from the company they keep) on a promised cliffhanger. Here it is.
Open a thesaurus brain and read two sentences:
"I deposited money at the bank."
"We walked along the river bank."
Under static embeddings, the word "bank" is a lookup: token ID → row of matrix E. The row is chosen by the spelling, not the sentence. Both occurrences fetch the identical vector — one frozen address for two completely different places.
That vector is a sense-average: a geometric compromise between unrelated meanings, living somewhere "between" finance and riverside, belonging to neither.
And polysemy is not a corner case:
- In WordNet terms, the most frequent English content words carry 5–20 senses each. The static model is blind to all but the corpus-dominant blend.
- Word order suffers equally:
"dog bites man"and"man bites dog"produce the same bag of vectors for any architecture that consumes pooled embeddings. One of the two is about dinner. The model cannot tell which. - Negation is invisible: "not good" sits next to "good".
So the question becomes:
What if a word's address... were not fixed?
What if "bank" got a different vector each time it appears, computed from the sentence around it?
Static vs. Contextualized Representations 🔀
Static vs. Contextualized Representations 🔀
The same token gets one frozen vector under static embeddings, but a different representation per sentence under contextual models — this single property ended the word2vec/GloVe era.
Unlock Topic #133: Contextualized Embeddings: Words Re-Read in Every Sentence
You are viewing a preview. The full in-depth technical walkthrough, worked derivations, and code notebooks for this concept, along with self-assessment quizzes, are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and actionable was this distributed systems breakdown?