Reasoning Models: o1, o3, and DeepSeek-R1
Reasoning models are trained — with reinforcement learning over long chains of thought — to write a private draft before answering: plan, check, backtrack, try again. OpenAI's o1 (Sept 2024) proved the class; DeepSeek-R1 (Jan 2025) reproduced it openly using GRPO and rewards that an auto-grader can verify, and showed the deliberation emerges even without any worked examples. This topic covers the recipe, the emergent "aha" behaviors, the distillation wave, and the very real failure modes.
01.The Problem: The First Guess Is Often the Wrong One
Ask a classic LLM a tricky math problem and it answers immediately.
Not because it is confident — because it has no way not to. Standard training (next-token prediction, then imitation-style fine-tuning) teaches one-pass answering: predict, commit, move on.
But you know how hard problems actually get solved. By humans, at least:
scratch paper. Plans. Wrong turns caught mid-way. "Wait — let me check that."
Before 2024, the best imitation of this was chain-of-thought prompting: write "think step by step" and hope the model produces useful reasoning text. It helped, but the model was reciting reasoning patterns it had seen in data, not being rewarded for reasoning well. Nobody had trained the loop itself: try → verify → revise.
So the question becomes
What if we trained a model to generate a long internal draft — and rewarded it only when the final answer was right?
That is a reinforcement-learning question: the reward signal ("correct!") arrives at the end, and the model must discover on its own that drafting, checking, and backtracking raises the odds of that reward. In September 2024 OpenAI showed it works, at scale. In January 2025 DeepSeek showed it works without any human-written reasoning examples at all — the behavior was latent, and RL dug it out. This topic is that story and its economics.
The Reasoning-Model Training Recipe (R1-style) 🧠
The Reasoning-Model Training Recipe (R1-style) 🧠
Cold-start long-CoT SFT, RL with verifiable rule-based rewards via GRPO, then rejection-sampled SFT and final multi-task RL — or R1-Zero's pure-RL variant that skipped the warm start and still emerged reflection.
Unlock Topic #218: Reasoning Models: o1, o3, and DeepSeek-R1
You are viewing a preview. The full in-depth technical walkthrough, worked derivations, and code notebooks for this concept, along with self-assessment quizzes, are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and actionable was this distributed systems breakdown?