Overfitting
A model overfits when it memorizes the training set instead of learning the process behind it: it aces the data it studied and fails on data it has never seen. The symptom is the train/validation gap. This topic gives you the diagnostic (learning curves), the four mechanisms that cause the gap (capacity, data, leakage, evaluation abuse), and the ranked list of remedies that actually work.
01.The Problem: The Student Who Aced the Practice Test and Failed the Real One
Imagine a student preparing for a math exam.
There is a practice test with 200 questions and an answer key.
Our student does not learn algebra. They memorize the 200 answers.
Practice test: 100%. Real exam (new questions, same subject): they crash and burn.
Nothing was "wrong" with the studying. The student optimized the wrong target: the specific questions, not the subject behind them.
A model can do exactly this.
You train on 2 million ad clicks. Training accuracy: 0.99. You celebrate.
Next Monday, live traffic: 0.71. The model quietly memorized those 2 million rows — their typos, their duplicate users, the datestamp patterns, the noise — instead of learning why people click.
So the question becomes
How do we know the model learned the process, and not the training set?
The answer drives everything in this topic:
- Compare training score vs fresh-data score → the gap.
- If the gap is large, work out which mechanism caused it — there are four, and they need opposite fixes.
- Then apply remedies in order of payoff, not novelty.
One sentence to keep in your pocket: the gap is overfitting; the level of the validation curve is what you actually ship.
Diagnose, Attribute, then Fix 🩺
Diagnose, Attribute, then Fix 🩺
The train-validation gap is the symptom; capacity, data quantity/quality, leakage, and evaluation abuse are the four distinct causes, each with a different remedy.
Unlock Topic #52: Overfitting
You are viewing a preview. The full in-depth technical walkthrough, worked derivations, and code notebooks for this concept, along with self-assessment quizzes, are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and actionable was this distributed systems breakdown?