Weight Initialization: Xavier & He

You've built a 50-layer network. You've written the training loop. You run it, and the loss... just sits there. Or goes to NaN. Or the predictions are the same value for every input.

Nothing in your code is wrong. The weights were.

Here's the plain-sentence setup you need: at step 0, the network's weights are random numbers you sample from some distribution — and that choice happens before any gradient has ever flowed.

So the question becomes

"Random small numbers, whatever" is not an answer — and at depth it's fatal. Think about what a 50-layer forward pass actually is (Topic 82): a product of 50 matrices:

a^(L) ≈ (W_L·φ'·…·W_1)·x

Every layer multiplies the signal's scale by some gain. A gain of 0.9, fifty times: 0.9⁵⁰ ≈ 0.005 — the signal is gone. A gain of 1.1, fifty times: 1.1⁵⁰ ≈ 117 — the signal is a roaring saturation mess. Only a gain of ≈ 1.0 per layer lets the input survive to the output and lets gradients survive back to the first layer.

Whether the product vanishes or explodes is decided almost entirely by the variance of the distribution you sampled weights from. Two researchers worked out the arithmetic: Glorot & Bengio (2010) and He, Zhang & Ren (2015). Their rules — Xavier and He — are the difference between "deep nets were impossible" and "deep nets are a library call."