Exploding Gradients: When Backprop Goes to Infinity
The mirror image of the vanishing gradient (Topic 85): if each layer multiplies the learning signal by more than 1, gradients grow exponentially until weights leap across space, numbers overflow, and your loss turns into NaN. Learn the fingerprint of a blowing-up run and the defenses: gradient clipping, normalization, careful init, and loss scaling.
01.The Problem: One Step, and Your Model Becomes NaN
You start a training run at night. The loss curve is beautiful.
At step 402:
- loss 2.3
- step 403: loss 18,000
- step 404: loss NaN
NaN means "Not a Number" — the computer literally ran out of valid arithmetic. The model is dead. Every weight is poisoned. Restart.
What just happened?
One gigantic update. The gradients on step 403 were so huge that the update rule θ ← θ − lr·∇θ hurled the weights across parameter space, the next forward pass overflowed the numbers, and everything after that is garbage.
This is the exploding gradient problem.
If you read Topic 85, you already know the machinery. There, the learning signal multiplied down to zero and layers froze silently.
Here the same signal multiplies up to infinity and destroys learning loudly.
Same math. Opposite direction.
Explosion Mechanics and Mitigations
Explosion Mechanics and Mitigations
Same product-of-Jacobians math as vanishing, opposite direction. The standard rescue is clipping (cheap insurance) plus normalization (structural cure).
Unlock Topic #86: Exploding Gradients: When Backprop Goes to Infinity
You are viewing a preview. The full in-depth technical walkthrough, worked derivations, and code notebooks for this concept, along with self-assessment quizzes, are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and actionable was this distributed systems breakdown?