Recurrent Neural Networks: Sharing Weights Across Time

Imagine reading a novel one word at a time — through a keyhole.

You see a word. You react to it. Then the page is destroyed.

At the next word, you have no record of the previous ones.

That is exactly how a plain feed-forward layer works. It maps one fixed-size input to one output and forgets everything. Show it the sentence "the cat that the dog chased ran" one word at a time, and word 6 cannot use word 2.

Topic 110 said sequence data lives or dies on order and on dependencies reaching far back. So a model for sequences needs one extra ingredient:

Add that, and you get a recurrent network: a loop instead of a one-shot layer. It is the simplest way to give a network memory, and for roughly a decade (2013-2017) it was the default architecture for text and speech.

The entire recurrence is three lines:

h_t = tanh(W_h·h_{t-1} + W_x·x_t + b_h)

y_t = W_y·h_t + b_y

Run them for t = 1..T and you have processed a sequence of arbitrary length. That is the whole forward pass of a vanilla RNN.

Unpack every piece:

• x_t ∈ R^{d_x}: the embedding of the input at time step t (the "token vector" from Topic 110).
• h_t ∈ R^{d_h}: the hidden state — the network’s running summary of the prefix x_{1..t}, i.e. everything seen so far (Topic 112 studies this summary in depth).
• W_h ∈ R^{d_h×d_h}: the recurrent matrix — the only pathway between time steps. It says "how the old summary morphs into the new one."
• W_x ∈ R^{d_h×d_x}: the input matrix — how the fresh token gets folded into the summary.
• y_t: the readout — a small linear layer turning the summary into a prediction.

The one thing to burn in: h enters the formula twice — the W_h·h_{t-1} term means the previous output is re-fed as input. That feedback arrow is the memory channel along time.

Make everything tiny: one input unit, one hidden unit. Take W_h = 0.5, W_x = 1, b = 0, and start from h_0 = 0.

Feed the token sequence x = [1, 0, 1]. The rule is h_t = tanh(0.5·h_{t-1} + x_t).

Step 1: h_1 = tanh(0.5·0 + 1) = tanh(1) ≈ 0.76

Step 2: h_2 = tanh(0.5·0.76 + 0) = tanh(0.38) ≈ 0.36

Step 3: h_3 = tanh(0.5·0.36 + 1) = tanh(1.18) ≈ 0.83

Now look at what happened to the first token, x₁ = 1:

• At h₁ it is fully present (0.76).
• At h₂ its contribution has been multiplied by 0.5 once.
• At h₃ it has been multiplied by 0.5 twice — its trace is about 0.19, out of a state now dominated by x₃.

One token fades geometrically. With a real W_h acting on hundreds of channels, the same arithmetic says: old information decays unless something protects it. Hold that picture — Topic 113 shows the gradient decays the same way, and Topic 114 is the invention that fixes both.

Conceptually, "unroll" the loop for T steps and the RNN becomes a T-deep feed-forward network whose layers all share one set of weights:

code
Copy
 t=1            t=2            t=3            t=T
 x₁ ─►[cell]─h₁─►[cell]─h₂─►[cell]─ … ─h_{T-1}─►[cell]─h_T─► y_T
        ▲           ▲           ▲                  ▲
        └── same W_h, W_x, b reused at EVERY step ─┘

One cell, copied along time. That sharing buys three things no per-position MLP can:

1. Any-length inputs: the parameter count never mentions T. Train on 20-token sentences, run on 2,000.
2. Statistical efficiency: one W_x is updated by every position of every training sequence — a vanishingly small parameter count (2–3·d_h·d_x floats) compared to flattening time into the input width.
3. Temporal translation invariance: whatever pattern the cell learns at step 3, it applies at step 900 — the convolution idea, but along the time axis.

And one cost comes with the same trick:

• The chain of T nonlinearities is also a chain of multiplications on the gradient path. Depth is exactly what backpropagation handles badly. Unrolling turns "recurrence" into "extreme depth along time" — setting up Topic 113.

import torch

def rnn_forward(xs, Wx, Wh, b, h0=None):
    # xs: [T, B, d_x] ; Wx: [d_h, d_x] ; Wh: [d_h, d_h] ; b: [d_h]
    h = h0 if h0 is not None else xs.new_zeros(xs.shape[1], Wh.shape[0])
    hs = []
    for t in range(xs.shape[0]):                    # the loop IS the recurrence
        h = torch.tanh(h @ Wh.T + xs[t] @ Wx.T + b) # h_t = tanh(W_h h_{t-1} + W_x x_t + b)
        hs.append(h)
    return torch.stack(hs)                          # [T, B, d_h] — one state per step

Carry one analogy through the rest of the phase: a note-taker listening to a lecture.

The note-taker is the RNN cell. The rule is:

1. Hear one sentence (that's x_t).
2. Read your single line of notes (that's h_{t-1}).
3. Rewrite the line: best one-sentence summary of "notes so far + new sentence" (that's h_t).

Everything the note-taker will ever say or answer is based on that one line — the original lectures are gone from memory except as summarized there.

This analogy pays off repeatedly:

• One notebook line = fixed-size hidden state: cheap, portable, but lossy (Topic 112).
• Same handwriting habits every sentence = weight sharing: one skill, reused at every step.
• The summary is rewritten, not appended = why old facts fade: each rewrite multiplies the past by W_h and squashes through tanh.
• Being quizzed on sentence 500 = BPTT: to learn what note-taking rule would have helped at sentence 6, credit must flow back through 494 rewrites (Topic 113).

When LSTM "gates" arrive, they are exactly this note-taker being given a highlighter and a filing cabinet — same notebook, but lines can now be protected from rewriting.

Unrolling also defines training. To learn from the whole sequence, backprop flows through the unrolled chain: gradients accumulate into W_h and W_x from every time step, and to influence the earliest tokens they must traverse T cell-to-cell derivatives (Topic 113 analyzes why this goes badly).

Practical regimes used from ~2013 onward:

• Truncated BPTT: cut the graph every 20-200 steps, carry the detached hidden state forward. Bounds memory and gradient path length — at the cost of "seeing" only local dependencies.
• Per-step losses for language modeling: predict x_{t+1} from h_t at every step — dense supervision, no separate decoder needed (the CharRNN recipe of Karpathy, 2015, and still how causal LMs are defined).
• Final-step loss for classification: read only y_T (the many-to-one shape from Topic 110).

Vanilla RNNs are excellent at local, quickly-decaying structure: morphology inside a word, prosody, short-horizon dynamics, character-level primers. Bengio’s group and Mikolov’s speech models showed they capture grammar-like regularities within a few tokens.

Their measured ceiling is dependency range: empirically, vanilla RNNs learn dependencies reliably out to roughly 5-10 steps and degrade toward chance beyond, because gradients connecting distant events vanish exponentially (Topic 113 — the 0.5-factor decay from the worked example, now applied to the learning signal itself). The fixes arrived in order:

↓ gating (LSTM/GRU, Topics 114-115) ↓ then full pairwise visibility (attention, Topics 118-120).

In 2024-2026, vanilla RNNs are mostly a teaching layer — but the recurrent paradigm itself is having a revival: RWKV, RetNet, and Mamba-style state-space models are re-deriving "constant memory per token" recurrence with modern gating, precisely for the streaming and long-context economics RNNs always promised. The note-taker’s one-line notebook, it turns out, is still unbeatable when the notebook must fit in a pocket.