Long Context Models (1M+ Tokens)

Start from the pain.

A model's context window is, in one plain sentence: how many tokens it can see at once — prompt plus generated answer — because every token runs attention over every other visible token.

In 2023 that budget was 4K–32K tokens. A novella is ~100K tokens. A mid-size code repo is easily 500K. Two hours of video, transcribed, is millions.

So teams did awkward surgery:

• Chop documents into 500-token pieces, embed each piece, retrieve the top-5 similar ones, and stuff them into a small prompt. That is RAG (retrieval-augmented generation) — and it fails whenever the links are unpredictable: the bug in file 7 was explained in file 203, and no keyword query will ever connect them.
• Summarize-and-carry-forward for long agent sessions — every summary silently deletes something.
• Give up on whole-corpus questions.

So the question becomes

Nice goal. But naively making the window bigger breaks three different things at once — and the entire history of long-context models (2023 → 2026, from 4K to claimed 10M tokens) is the story of fixing all three together. That, and the uncomfortable gap between "the window accepts 1M tokens" and "the model uses 1M tokens."