ReLU: max(0, x) and the Deep Learning Explosion

Topic 77 in one plain sentence: activation functions are the tiny per-number rules between matrix multiplies, and the old favorite — sigmoid — flattens ("saturates") for large inputs, so its slope crashes toward 0.

Now put that in a deep network and run the arithmetic.

Backprop multiplies one local slope per layer on the way back to the early weights.

Sigmoid's best-case slope is 0.25. Its typical trained slope is far worse.

code
Copy
20 layers at best case:  0.25^20  ≈ 10^-12   ← a trillion-fold attenuation
20 layers at |z| ≈ 5:    0.0066^20 ≈ 10^-45  ← gradient physically gone

A 1990s engineer watching this sees:

• the last layers learn (their gradients are fine),
• the first layers never move — the signal reaching them is smaller than floating-point noise,
• so "just add more layers" is a death sentence. Depth was the promised land, and sigmoid networks could not reach it.

So the question becomes

The 2010 answer was so simple people laughed.

ReLU (Rectified Linear Unit), popularized by Nair & Hinton's 2010 work on Restricted Boltzmann Machines and cemented by Krizhevsky's 2012 AlexNet, is:

code
Copy
φ(z)  = max(0, z)
φ'(z) = 1 if z > 0, else 0

Two lines later we will see why five words changed the field.

ReLU is simply

That is a gate, not a squeezer. Compare what happens to an increasingly confident input:

code
Copy
z:        0.5    2     10     100     1000
ReLU:     0.5    2     10     100     1000     ← passes through FOREVER
sigmoid:  0.62   0.88  0.9999 ~1.0    ~1.0     ← pinned at 1; slope ~0

Unpack the properties, one plain sentence each:

• Unbounded above. No ceiling means no shoulder to get stuck on — there is no z where ReLU goes flat on the positive side.
• Slope exactly 1 when active. The gradient doesn't shrink at all passing through an active ReLU — it multiplies by 1, which changes nothing.
• Slope exactly 0 when inactive. Negative scores emit nothing and backprop nothing. Clean, absolute silence — which is both the feature (sparsity) and the bug (death; section 5).
• Cheap. A comparison and a copy. No e^-z, no division, no table lookup.

Read z as the pre-activation w·x + b — the perceptron's score from topic 75, the thing a hidden unit computes before its activation.

Send the same gradient through three layers twice — once with sigmoids, once with ReLUs — and watch the numbers.

Suppose layer 3 produces an error signal of 1.0 and we backpropagate it.

Journey A — sigmoid layers at realistic operating points z = 5, 3, 1:

code
Copy
σ'(5) ≈ 0.0066 → gradient: 1.0 × 0.0066        = 0.0066
σ'(3) ≈ 0.0451 → gradient: 0.0066 × 0.0451     = 0.0003
σ'(1) ≈ 0.1966 → gradient: 0.0003 × 0.1966     = 0.000058

By layer 1, the correction is 58-millionths of its original size. Layer 1 barely hears it.

Journey B — ReLU layers, inputs z = 5, −3, 2 (one unit inactive):

code
Copy
ReLU'(5) = 1  → gradient: 1.0 × 1   = 1.0
ReLU'(−3) = 0 → gradient: 1.0 × 0   = 0   (that ONE path is silenced)
ReLU'(2) = 1  → gradient: 0.0 × 1   = 0

Two lessons in the arithmetic:

1. On active paths the magnitude is untouched — 1.0 arrives at layer 1 as 1.0. No drowning. Depth is suddenly survivable.
2. The inactive unit contributes exactly zero — not "small", but nothing. Now imagine the inputs to one unit are negative for every example in the dataset... (that is section 5's corpse).

Forward pass with concrete scores, for completeness:

code
Copy
z = ( 2.0, −5.0, 0.3, −0.1 )  →  ReLU →  ( 2.0, 0.0, 0.3, 0.0 )
                                    ↑ 2 of 4 units fired = 50% sparsity,
                                      exactly the typical trained rate

Draw the two functions on the same axes and the story is visible:

code
Copy
 ReLU:  ramp that never tops out        sigmoid/tanh: S-curve with FLAT shoulders
 output                                output
   6 ┤                  ╱                1 ┤              ─────── ← flat: slope ≈ 0
   4 ┤              ╱                    .5┤          ╭───╯
   2 ┤          ╱                          │      ╭──╯   slope ≤ 0.25
   0 ┤─╱ ← wall: slope EXACTLY 0           └──────╯
   └───────────────────────► z              ╲ flat: slope ≈ 0
       left of 0: out 0, grad 0                    ─────────────► z
       right of 0: out = z,  grad = 1              6 ┤ ─────── 1.0 sigmoid
                                          (tanh: same shape, −1..1, center 0)

Better, simpler: ReLU is a check valve on a water pipe.

code
Copy
   positive flow ──►►►► [ valve ] ──►►►►  full flow, undiminished
   negative flow ◄──── [ valve ] ✕       blocked completely, 0

Water = both the signal going forward and the gradient coming backward. The valve:

• never throttles the allowed direction (sigmoid does — that's why deep sigmoids starve),
• but blocks the other direction absolutely — and a pipe that's permanently blocked has no reason to ever be cleaned (dead unit, section 5).

Three diagrams in one: (a) the ramp explains no-saturation, (b) the flat-left explains sparsity, (c) the flat-left's zero slope explains dying. The same geometry is the feature and the bug.

A ReLU unit is dead if its pre-activation z = w·x + b goes negative for every training input.

What happens mechanically:

• Gradient w.r.t. a dead unit's weights is exactly 0 (chain rule hits the 0 branch).
• With large steps, one bad update can push b below the entire input cloud of that unit. Picture the doorman from topic 75 slamming his bar so high that literally nobody in the building can reach it.
• Dead units stay dead because nothing in the loss can resurrect them: zero gradient → no future update can ever touch those weights again. Not "slowly fading" — permanently frozen, mid-stride.

A network can lose a substantial fraction of capacity this way silently — the loss curve just plateaus early and nobody knows why.

The standard mitigations, matched to the valve story:

1. He initialization — variance-aware init that fires ~half the units at step 0. (Fresh pipes open at the factory with water already flowing.)
2. Lower learning rates / gradient clipping — stop one huge update from slamming the gate. (Don't crank the pressure wrench past the rating.)
3. Leaky ReLU — max(0.01z, z) gives the negative side a small slope so gradient always exists. (A pinhole drip through the closed valve: enough to keep the mechanism loose.)
4. PReLU — learns the slope per unit (He et al., 2015 — beat ReLU on ImageNet-scale nets with learned slopes). (Each pipe engineers its own drip rate from experience.)

Detection is cheap — see the callout below. Prevention is architecture hygiene.

Once you accept "the left side is the problem", the patches write themselves — each one is a different policy for what happens below zero:

code
Copy
 output
   │        ╱ ReLU        hard wall:   slope 0
   │       ╱  Leaky      thin drip:    slope 0.01
   │      ╱·PReLU        learned drip: slope α (per channel)
   │     ╱                RReLU:       random drip in [l,u],
   │    ╱╲_____ ELU                       fixed mean at test
   │___╱       ‾‾‾‾‾____  SELU: ELU with self-normalizing constants
   └──────────────────────────────► z

• Leaky ReLU: max(αz, z), α=0.01. Fixes dying; keeps sparsity nearly intact. Dominant in GAN discriminators.
• PReLU: α learned per channel (backprop's chain rule through the minimum — He et al. gave the elegant derivation: the gradient flows to α through whichever branch wins the min). Risk: with learned α > 1 it can invert/absorb gradients.
• RReLU: α random in [l, u] during training, fixed mean at test — a built-in regularizer.
• ELU: z if z>0, α(e^z − 1) if z≤0 — smooth negative saturation, closer to zero-centered outputs, at exponential cost.
• SELU: ELU with fixed scale 1.0507/α=1.6736 that self-normalizes under strict assumptions (now mostly replaced by BatchNorm/LayerNorm).

And a deeper reason ReLU beat its predecessors that goes beyond one unit's gradient: a loss-landscape study (Li et al., 2018, "Visualizing the Loss Landscape of Neural Nets") showed ReLU networks suffer far fewer bad local minima than sigmoid/tanh ones.

Why? Scale-invariance of ReLU weights: because max(0, z) has no ceiling, multiplying a unit's weights by any positive constant c leaves its decision unchanged (ReLU output scales by c, and downstream layers absorb it) — equivalent minima connect and the landscape flattens and becomes navigable, while saturating units create chaotic high-curvature regions where the "ground" itself flickers.

The family in runnable form:

import numpy as np

relu   = lambda z: np.maximum(0, z)
leaky  = lambda z, a=0.01: np.maximum(a * z, z)
prelu  = lambda z, a: np.maximum(a * z, z)     # a is a learned tensor
elu    = lambda z, a=1.0: np.where(z > 0, z, a * (np.exp(z) - 1))

# dead-unit audit inside a training loop (PyTorch):
# frac_zero = (pre_activation <= 0).float().mean()
# if frac_zero > 0.8: print("layer is half-dead — lower LR")

One geometric consequence of ReLU deserves its own mental model, because it powers half of modern interpretability research.

A ReLU network is a piecewise-linear function: each input region maps linearly, and the network traces regions via the on/off pattern of units — the geometric viewpoint behind modern "deep learning is piecewise linear interpolation" analyses.

Feel it with one unit in 2-D:

code
Copy
h = max(0, x1 + x2 − 1)

  x2
   ▲    ╲╲╲  z>0: output = x1+x2−1 (one flat plane)
   │   ╲╲╲
   │  ╲╲╲╲╲        ╱ = the crease where z = 0
   └────────╲─────► x1
      z<0: output = 0 (flat floor)

One unit: two tiles. Ten thousand units: a mosaic whose tile pattern is chosen by the data the unit saw.

• Every distinct on/off firing pattern = one tile = one little linear model.
• Training bends creases so the tiles align with the task.
• Counting tiles is exactly what Montufar-style "linear regions" analysis (topic 76) does — with ReLU, depth multiplies tiles exponentially.

This is also why monitoring frac_zero (the code block above) reads like diagnostics of the mosaic: you are literally counting how many units stopped contributing tiles.

Transformers moved to GELU/SiLU (topic 77), but nobody evicted ReLU:

• It remains the default hidden activation in CNN vision backbones (ResNet, MobileNet, EfficientNet variants), GAN discriminator heads, RL policy/value nets, and every fast prototype.
• Hardware keeps reinforcing it: NVIDIA's ReLU/MAX operations map to single ALU instructions, and ReLU-only networks are the native diet of quantized inference — integer max is trivial, while exponentials for GELU approximations cost cycles on tiny NPUs (see the real-world example below for the Apple/Qualcomm edge story).

And the intellectual legacy matters most:

Read that sentence twice. The 1990s sigmoid nets could represent the functions we wanted — universal approximation was proven in 1989-91. What they could not do was route gradient back to their own front door. Fix the plumbing, and the same architectures suddenly trained — AlexNet 2012, then everything after. The insight that unlocked 2012-2025 deep learning is not "we found a better function class"; it is "we found a function whose slope doesn't die".

Quick self-check to make sure you own this topic:

1. Why does an active ReLU transmit gradients without attenuation? (φ' = 1 exactly.)
2. Why is a permanently-inactive unit frozen, not merely quiet? (Zero gradient → zero weight update → forever.)
3. Which two costs does Leaky ReLU trade a 1% slope for? (Death insurance + slightly less sparsity.)
4. Why do edge NPUs still prefer ReLU over GELU? (Integer max, no exp, fused kernels.)

If all four answer instantly, you understand why one max outlived three research generations.