Weight Decay: L2 Regularization, Decoupled
Weight decay gently pulls every weight toward zero at each step, favoring small, smooth solutions. For SGD it is exactly L2 regularization; for Adam it is not — AdamW fixes that by decaying the weights directly. This topic covers the math, the AdamW correction, and the no-decay-on-norms convention.
01.The Problem: Big Weights Memorize Noise
A network fits data by tuning its weights. Give it enough freedom and it will do something you do not want: it will grow huge weights that twist the function into sharp knots, memorizing random noise in the training set instead of learning the smooth real pattern.
Huge weights make the function jumpy. A tiny change in input then causes a wild change in output — that is overfitting, and it shows up as "great on training data, terrible on new data".
So the question becomes:
Can I gently punish the model for using large weights, nudging it toward simpler, smoother solutions?
Yes. The oldest trick in the book is weight decay: at every step, drag every weight a little bit toward zero. Only weights that clearly earn their keep — by lowering the loss enough to fight the drag — stay large.
Coupled vs Decoupled Decay
Coupled vs Decoupled Decay
Under SGD the two are identical. Under Adam-family optimizers, gradient-side decay is rescaled by the adaptive denominator; AdamW multiplies weights directly, restoring intent.
Unlock Topic #94: Weight Decay: L2 Regularization, Decoupled
You are viewing a preview. The full in-depth technical walkthrough, worked derivations, and code notebooks for this concept, along with self-assessment quizzes, are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and actionable was this distributed systems breakdown?