GPT: The Generative Pre-training Lineage
The GPT lineage is one recipe — predict the next word — pulled through six upgrades: a decoder pre-trained then fine-tuned (GPT-1, 117M), zero-shot at 1.5B scale (GPT-2), few-shot prompting at 175B (GPT-3), human alignment (InstructGPT), multimodality (GPT-4/4o), and test-time reasoning (o1-o3). This topic walks how each step changed one lever and turned a language model into the default foundation for chat assistants, agents, and code generation.
01.The Problem: A Reader Is Not a Writer
You now know BERT: a champion reader (Concept 134) that can never write a sentence.
Fill in the blanks with it? Perfect.
Ask it to compose an email? Silence.
What if we trained the other half of the Transformer instead — the part that produces text one token at a time?
That is where the GPT line begins. And the task it must solve is the most natural one in language:
given everything said so far → what word comes next?
Sounds trivially small. The whole lineage — from a 117M-parameter experiment in 2018 to the reasoning models of 2025 — is the story of how that one question, asked at internet scale, turned out to contain almost everything else.
One rule to hold onto for the entire topic: the objective never changed. Every generation changed something around it — data, size, alignment, senses, thinking time. Nothing more, nothing less.
The GPT Lineage Timeline 🚀
The GPT Lineage Timeline 🚀
Each step changed the dominant lever: architecture (GPT-1), data and scale (GPT-2/3), alignment (InstructGPT), modality and inference-time compute (GPT-4o/o-series).
Unlock Topic #137: GPT: The Generative Pre-training Lineage
You are viewing a preview. The full in-depth technical walkthrough, worked derivations, and code notebooks for this concept, along with self-assessment quizzes, are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and actionable was this distributed systems breakdown?