LSTM: Gates That Rescue the Cell State

Topic 113 ended with a stark picture: gradient credit for a distant event must pass through a product of T copies of diag(tanh′)·W_h, and since the per-step gain γ is essentially never exactly 1, credit either fades to nothing (γ < 1) or detonates (γ > 1).

The whisper-game diagnosis from last time suggested the fix directly:

Translated into math, "full volume" means a self-loop whose derivative is exactly 1.0:

∂C_t/∂C_{t-1} = 1

No matrix multiplication per step. No tanh squashing per step. Just the old value, preserved, unless something chooses to change it.

Hochreiter & Schmidhuber’s 1997 insight (built on Schmidhuber’s 1992 "learning to control fast-weight memories") was to stop routing memory through a repeatedly-multiplied matrix and instead give the cell a self-loop that can be held at exactly 1.0 — with a learned dial controlling how close to 1 it stays.

The LSTM gives the cell a second, special memory vector — the cell state C_t — that runs through time like a conveyor belt. Only two things may touch it, and both are gentle multiplications:

∂C_t/∂C_{t-1} = diag(f_t)

where f_t is the forget gate: a learned number between 0 and 1 per channel.

Read that derivative out loud:

• If the gate settles near f_t ≈ 1, the state — and the gradient riding on it — passes through unchanged.
• If f_t ≈ 0, that channel is deliberately wiped.
• The gradient product over T steps becomes ∏ diag(f_t) ≈ I when gates stay open: error "carousels" back undiminished.

This is the constant error carousel, and it is the whole theoretical trick of the LSTM: not a cleverer activation, but an additive, gated identity path.

The one-line version:

One channel holds a fact — say the entity "Priya" from step 100 tokens ago, stored as C = 1.0.

Vanilla RNN (Topic 113): every step multiplies credit by gain ≈ 0.9.

• 0.9^100 ≈ 0.00003 — the learning signal about "Priya" never arrives. Game over.

LSTM: every step multiplies by the forget gate instead. Two scenarios:

• Gates stay open at f = 0.99: 0.99^100 ≈ 0.37. Thirty-seven percent of the original credit still reaches step 1. That is enough to learn from.
• A new, unrelated sentence starts: the gate slams to f = 0.1 on that channel → the old fact is erased in one step, freeing the ledger.

And the content survives the same way: C_t = 0.99·C_{t-1} + (small write) keeps 1.0 at about 0.37 after 100 steps rather than the double-decay a vanilla state would suffer.

0.99 vs 0.9 is the difference between a memory of a few dozen steps and a memory of hundreds — and the model itself picks the base, per channel, per step. That is the entire philosophy of gating in two lines.

With [h_{t-1}, x_t] the concatenation (all gates see both):

1. Forget: f_t = σ(W_f·[h_{t-1}, x_t] + b_f) — what fraction of old memory survives (per-dimension, learned).
2. Input: i_t = σ(W_i·[h_{t-1}, x_t] + b_i) — how much new information is written.
3. Candidate: C̃_t = tanh(W_C·[h_{t-1}, x_t] + b_C) — the proposed new content.
4. State update: C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t — the protected ledger: erase, then append.
5. Output: h_t = o_t ⊙ tanh(C_t), with o_t = σ(W_o·[h_{t-1}, x_t] + b_o) — a filtered read of the ledger.

Sanity checks that reveal understanding:

• σ gates emit values in (0,1) — they are mixers, not switches.
• ⊙ is elementwise, so each of the d_h channels keeps its own schedule: one channel can forget while its neighbor remembers.
• h_t can differ arbitrarily from C_t — what the cell knows versus what it shows are separate resources.

Picture the cell state as a horizontal conveyor; everything else happens around it:

code
Copy
        f_t (keep 0.99?)      i_t (write 0.05?)        o_t (show 0.8?)
 C_{t-1} ────×─────────── + ────────×──── ──────► C_t
    ═══════════╡ LEDGER HIGHWAY ╞══════════════════►  (never touched by W_h)
                                ▲
                          C̃_t = tanh(...)   (new content proposed)

 C_t ──► tanh(C_t) ────×────► h_t ──► prediction, and into the next [h, x]
                       ▲
                     o_t (reading lamp: what to reveal)

• The top lane is the gradient highway of Topic 113: a product of gate values that can stay near 1.
• Below it, h_t is just a spotlight aimed at whichever parts of the ledger matter right now.

If you remember one picture from this topic, remember the highway: memory moves along an additive line; matrices only ever decide the gate dials.

Continue the whisper-game cast from Topic 113. The office now keeps an accounting ledger (that’s C_t) and a daily report (that’s h_t). Three gatekeepers stand around the ledger, and all three read the same two things: yesterday’s report and today’s mail ([h_{t-1}, x_t]).

• Eraser (forget gate): decides how much of each old line to keep. Most days he barely touches anything (f ≈ 0.99); at a chapter break he wipes a page clean (f ≈ 0).
• Clerk (input gate): writes today’s new fact (candidate C̃_t) onto the ledger — the amount written is scaled by his dial.
• Spokesperson (output gate): reads the ledger aloud as the daily report, revealing only the lines relevant to today (h_t = o_t ⊙ tanh(C_t)).

Because the ledger itself is never carried forward through the multiplying machinery. Pages simply persist unless the Eraser or Clerk edits them — so an auditor tracing a decision back 500 days (backprop) finds the entries legible, thanks to .99^500 being a slow fade, not a cliff. The 1997 inventors literally drew the conveyor as a carousel; the whisper line finally got its loyal repeater.

Follow one gradient on the BPTT path from C_T to C_1: it passes through diag(f_T)⋯diag(f_2) — no W_h, no saturation factor. If gates learned to stay open on the channels holding a relevant fact, the gradient arrives undamped. Learning is precisely the discovery of when to keep gates open: subject-verb agreement across a clause, an entity’s number, a quote’s open delimiter.

What remains unfixed:

• Range is bounded, not infinite: gates drift from 1, so faithful memory is empirically ~hundreds of steps, not 100k; exact-copy tasks still degrade (Topic 112).
• One path per pair of events: an LSTM can hold a few salient threads, not a searchable index of everything.
• Exploding gradients still occur on input/weight paths — keep the clipping.
• No parallelism across time: per-step cost went 4× higher; the sequential bottleneck got worse, not better.

import torch, torch.nn as nn

class MiniLSTM(nn.Module):
    def __init__(self, d_in=300, d_h=512):
        super().__init__()
        self.W = nn.Linear(d_in + d_h, 4 * d_h)   # f, i, C-cand, o in one shot
        self.d_h = d_h
    def forward(self, x, state):
        h, c = state
        f, i, g, o = self.W(torch.cat([h, x], -1)).chunk(4, dim=-1)
        c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)   # ledger
        h = torch.sigmoid(o) * torch.tanh(c)                          # filtered read
        return h, (h, c)

Transformers took NLP (Topic 124), but LSTM-shaped recurrence kept the niches where its properties are unmatched:

• Streaming ASR/TTS and keyword spotting: constant per-step latency and a few-KB state beat KV-cache growth on hearing aids, wearables, and on-device wake-word models; bidirectional/projection-LSTM encoders remain common in telephony stacks.
• Time-series forecasting at scale: Amazon’s DeepAR (an RNN likelihood model) still powers demand forecasting fleets where per-request memory must be flat.
• Reinforcement-policy memories and hybrid agents with explicit state carryover.
• As teaching: the gate equations are the ancestor of every 2024 recurrent revival — Mamba’s selection mechanism and RWKV’s time-mixing are formally "forget/input gates with learned, input-dependent decay."