Backpropagation: The Chain Rule, Engineered

You know the forward pass (previous topic): numbers flow input → layers → prediction. And you know the goal of training (gradient topics): nudge every weight a tiny bit downhill on the loss, using θ ← θ − lr·∇L.

So the question becomes

For a network with 10^8 parameters, that's one hundred million partial derivatives. Per training step. Before backprop existed, you had two ideas, and both are disasters:

• Numerical differentiation (brute force): nudge one weight, re-run the forward pass, see how the loss moved. That's ~2 forward passes per parameter — about 10^8 forward passes per step. A single update would take longer than the heat death of the universe. Absurd.
• Symbolic differentiation: ask math software to write one giant closed-form expression for ∂L/∂w across a 50-layer net. The expression explodes combinatorially, and every shared sub-computation gets rewritten and recomputed dozens of times.

The insight that saved deep learning: the network is a composition of simple scalar/tensor ops, each with a trivial local derivative. If you exploit that graph structure, you get all gradients in about 2× the cost of one forward pass — no matter how many parameters share it.

Historically the algorithm was published for neural nets by Linnainmaa (1970), rediscovered by Werbos (1974), and popularized by Rumelhart, Hinton & Williams (1986). It is called backpropagation.