GRU: A Leaner Gate Design

Read Topic 114’s shopping list and you might wince:

• two state vectors (h_t and C_t)
• three gate matrices plus a candidate matrix — 4× the weights of a vanilla RNN
• one extra gate (forget) that only arrived two years after the paper

Every piece had a justification. But engineering asks the follow-up question:

In 2014, Kyunghyun Cho and colleagues ran exactly that audit for their machine-translation research. The result was the Gated Recurrent Unit (GRU): the LSTM’s gradient cure, rebuilt with one vector and two knobs. This topic is the audit — what got merged, what got thrown out, and what each choice costs.

Cho et al. (EMNLP 2014) asked: does the LSTM need C_t and h_t as separate resources, plus three gate matrices besides the candidate? Their answer: the Gated Recurrent Unit — one state vector, two gates:

z_t = σ(W_z·[h_{t-1}, x_t]) (update gate)

r_t = σ(W_r·[h_{t-1}, x_t]) (reset gate)

h̃_t = tanh(W_h·[r_t ⊙ h_{t-1}, x_t]) (candidate)

h_t = (1 − z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t

Read the last line first — it is the whole cell:

• The update gate z_t plays LSTM’s forget+input pair jointly: z_t ≈ 0 keeps the old state (remember), `z_t ≈ 1) fully rewrites it (forget-then-write, in one motion).
• The reset gate r_t protects the candidate from irrelevant ancient state — the same job as the forget gate, but applied before mixing, on the input side: r_t ≈ 0 means "propose today’s content from scratch, ignoring history."

And the gradient story survives: ∂h_t/∂h_{t-1} contains the additive (1 − z_t) ⊙ I term, so open-update-gate channels again give a near-identity carousel. GRU inherits LSTM’s core fix with fewer moving parts.

The state is one number. The update rule: h_t = (1−z)·h_{t-1} + z·h̃_t.

Scenario 1 — ordinary step. Old state h_{t-1} = 20. A new reading arrives, candidate h̃_t = 30. Gate learns z = 0.3:

• h_t = 0.7·20 + 0.3·30 = 14 + 9 = 23

A smoothed running estimate — classic exponential moving average, with a learned smoothing dial.

Scenario 2 — chapter break. A new paragraph starts; the old fact is now noise. The cell sets z = 0.9 and candidate h̃ = 5:

• h_t = 0.1·20 + 0.9·5 = 2 + 4.5 = 6.5 — nearly rewritten in one step.

Scenario 3 — holding a long-range thread. To carry a fact 100 steps untouched, the cell holds z = 0.01 the whole time:

• gradient credit multiplies by (1−0.01) per step: 0.99^100 ≈ 0.37 — 37% of the credit survives the trip (compare vanilla’s 0.9^100 ≈ 0.00003, Topic 113).

The same three behaviors (blend, rewrite, preserve) cost the LSTM two gates and two vectors; the GRU gets them from one dial. What it no longer has: a place to store a fact without it showing up in the readout — the price the missing output gate charges (Section 5).

Compare the two cells as plumbing. The LSTM runs a separate ledger highway with three toll booths. The GRU collapses it:

code
Copy
 LSTM:    C ledger ═══════╡f╞══════╡i╞══════► C_t ──tanh──╡o╞──► h_t
                           erase    write           read

 GRU:     h ══════╡(1−z) keep ╞══════╡z renew ╞═══════► h_t ──► y_t AND h_{t+1}
                             ╠══ r (reset): filters the past ENTERING the candidate

• ↓ The old state flows forward scaled by (1−z) — the gradient highway, now the only state.
• → The candidate flows in scaled by z — one motion does erase-and-write.
• → The same vector serves as memory and report. Nothing is held in silence.

If the LSTM is a library with a staff-only archive (C) and a display shelf (h), the GRU is a single desk: whatever you know is what’s on the desk, and the desk is what visitors see.

Continue the cast from Topic 114’s ledger-and-gatekeepers office. The GRU is the startup version of the same firm:

• One notebook instead of ledger + daily report. What is written is the report — the same desk doubles as archive and front counter.
• The Renewal Clerk (update gate z): one knob doing two jobs — "how much of today’s page survives into tomorrow" and "how much of the new memo gets written now." Most days the clerk barely touches the page (z ≈ 0.01); at chapter breaks the clerk rewrites the whole page.
• The Bouncer at the Proposal Desk (reset gate r): when the writer drafts the candidate, the bouncer decides how much of the old notebook is allowed to influence the draft. r ≈ 0 = "start fresh; yesterday’s gossip must not contaminate this sentence."
• The spokesperson is gone: no Output Gatekeeper — because there’s no archive to guard. Everything on the desk is shown.

Quiet knowledge. The big firm can keep a page in the archive while showing a different face at the counter; the startup’s memory leaks into every conversation simply by being on the desk. On tasks where facts must sit silently for hundreds of steps while other facts chatter, that coupling measurably hurts (Section 6). On tasks where remembering is saying, the two-keeper office is cheaper and just as good.

Benchmarked across Jozefowicz et al. (2015) and a decade since:

• Parameters: GRU has 3 recurrent weight blocks (W_z, W_r, W_h) vs LSTM’s 4 — 25% fewer recurrent parameters, and no second state vector (h only, not (h, C)).
• Small data / short sequences: GRU typically ties or edges LSTM; fewer parameters mean less overfitting and faster convergence.
• Long-range, memory-heavy tasks: LSTM (especially with peepholes or layer-norm) tends to win when the task demands holding several interleaved threads — the separate protected ledger buys real capacity.
• Speed on CPUs/edge: GRU’s smaller state matrix and single-vector carry often translate to 10-20% lower per-step latency; streaming ASR and wake-word models frequently ship GRUs for this reason.
• Quality at NLP scale: neither dominates decisively — the field’s verdict was "pick by budget," then both lost NLP to Transformers anyway (Topic 124).

# nn.GRU(hidden_size=512, num_layers=2) returns h only (vs nn.LSTM’s (h, c))
# Per-step cost: 3 · d_h · (d_h + d_x) recurrent MACs,  LSTM: 4 · d_h · (d_h + d_x)
# Same interface for streaming: carry h across chunks, detach under truncated BPTT.
# Swap is nearly one-line: torch.nn.GRU(x, h)  <->  torch.nn.LSTM(x, (h, c))

LSTM’s separation of store (C_t) and display (h_t) is not free in GRU: its single vector is simultaneously the memory and the readout. Consequences:

1. Coupling pressure: channels that must stay stable (long memory) also keep emitting, shaping every downstream step; LSTM can hold silent knowledge.
2. Interference rises on tasks multiplexing many facts: empirical probes (e.g., adding a small output gate, "outGRU," 2016) recovered much of LSTM’s edge on algorithmic/narrative memory tasks.
3. Where it does not matter: most control-style or single-thread tasks (sensor state tracking, simple dialog trackers) never stress the separation.

Rule of thumb that has held for a decade:

The interesting fact about GRU in 2024-2026 is that its design pattern outlived its rivalry with LSTM:

• Highway networks (2015): the (1−z)⊙a ⊕ z⊙b interpolation is the same algebra, applied per-layer instead of per-step.
• SRUs / "Recurrent Residual" nets (2017-2021): decoupled update gates that restore time-parallel training while keeping constant-size state.
• RWKV-4/5/6 (2023-2025): the time-mix channel is literally a learned, input-dependent decay (forget-style gate) over an exponential moving sum — GRU thinking retooled for Transformer-scale compute.
• Mamba / SSM selection (2024): the Δ parameter gates how much state renews per token — the update gate reborn as a continuous-time discretization knob.

When interviewers ask "why should I know GRU in the LLM era," this is the answer: gated recurrence is the algebra of learned memory, and the current linear-attention wave is its revival at scale.