GloVe: Global Vector Representations from Co-occurrence

Recall word2vec (previous concept, in one plain sentence: a tiny network plays "guess the neighbor word", and the learned profiles become word vectors).

It works. But notice how it sees the corpus: one window at a time. Each training sample is a single center-context pair. Somewhere in the statistics of billions of pairs, there is a giant fact about language that word2vec only ever implies:

Word2vec walks past this table 30 billion times without ever reading it.

Meanwhile, the older count-based methods of the 1990s–2000s — LSA, HIPPIE, PMI matrices — did read the whole table. They had global structure to spare. But they produced dense, non-geometric similarity scores: no compact vectors, no king−man+woman arithmetic, nothing you could hand a neural network as a feature.

Two camps, each missing the other's superpower:

code
Copy
   predictive (word2vec)      count-based (LSA/PMI)
   ✔ nice vectors             ✔ sees the WHOLE matrix
   ✔ geometry works           ✘ no usable vector space
   ✘ sees one window/step     ✘ no training signal
             ↘            ↙
              GloVe wants BOTH

GloVe's thesis paper (EMNLP 2014, Pennington, Socher & Manning) announced its ambition in three words: "counts, but also gradients."