Sequence Data: Why Order Carries Meaning

Start with a spreadsheet.

It lists 1,000 customers: age, income, purchases. Each row is one independent example.

Now shuffle the rows. Swap row 7 with row 900.

Did anything change?

For most models — a random forest, say — no. Tabular data is permutation-invariant: you can rearrange the training rows and the model does not care.

Now try the same trick on a sentence.

"dog bites man"

Shuffle the words:

"man bites dog"

Identical multiset of tokens. Opposite events. The first is an accident; the second is news.

Or a stock-price series:

1, 2, 3, 4, 5 — a calm rally.

5, 4, 3, 2, 1 — a crash.

Same five numbers. The order carries the entire meaning.

This is the dividing line for everything in Phase 6:

Text, speech, sensor readings, clickstreams — in all of them, order is the signal. A model that treats them like a bag of shuffled rows throws away exactly the information you wanted.

Sequence data has three properties. Each one is a problem for the fixed-input models you met earlier.

1. Order dependence.

Position matters. The prediction at position t must be conditioned on what came before (or after, in bidirectional settings). "dog bites man" ≠ "man bites dog" — the model has to know who came first.

2. Variable length.

Sentences run from 1 word to 500. Audio clips from 0.2 s to hours. A plain MLP has a fixed-size input layer — it cannot even accept a batch where one example has 3 items and the next has 41.

3. Dependencies span distance.

The referent of a pronoun may sit 2 or 200 tokens back. A seasonal pattern in energy demand recurs every 24 hours — thousands of samples away. Related events can be very far apart, and the model must still connect them.

Sequence models are the family of architectures built specifically to absorb these three constraints:

• RNNs compress the past into a running state that updates one step at a time (Topic 111).
• Transformers let every element look at every other element directly (Topics 119-120).

So the question that drives this whole phase becomes

This topic sets the vocabulary; the next ten topics are the machinery.

A temperature sensor reports four hourly readings, in order:

[10, 12, 14, 16]

Question: what happens next?

• Read in order: it rises about +2 degrees per hour. Guess 18. Something is warming up steadily.
• Shuffle them: [13, 10, 16, 11]. Same four numbers. Now there is no trend at all — just noise.

Sanity-check with arithmetic:

• Order-blind summary (a "bag" of numbers): the mean is 13 in both series, the spread is similar. The two datasets look identical.
• Order-aware summary: the first series has slope +2/hour (10→12→14→16); the second zig-zags (13→10→16→11), slope ≈ 0. The trend — the thing you actually care about — exists only in the ordering.

Words behave the same way:

"The shop closed because it was late."

You can only answer by remembering that "shop" came two tokens earlier. Scramble the sentence and the pronoun floats free — the meaning is gone even though every word is still present.

Every sequence model you meet from here on is a machine for preserving exactly this: what came before, and how far back it was.

Whatever the modality — text, audio, events — the input passes through the same four stations before a model ever sees it:

code
Copy
 raw input        tokenize            embed             pad + mask
"the cat sat" →  [101][569][2368] →  3 vectors     →   rectangular batch
 25 ms audio  →  [frame][frame]  →   of size d     →   [batch × T × d]
 view→cart→buy → [evt3][evt7]    →   (a lookup)    →   + 1/0 mask

Inside one batch, sequences have different real lengths, so short ones get fake filler tokens (PD) up to the longest, and a mask marks what is real:

code
Copy
 sentence A (4 tokens):  a1  a2  a3  a4        mask: 1  1  1  1
 sentence B (2 tokens):  b1  b2  PD  PD        mask: 1  1  0  0
                                                   ▲ 0 = "ignore me"

Read the sketch the way the GPU does:

• ↓ The model consumes one station at a time, left to right, order preserved.
• → Every row must be the same width, so padding equalizes the shapes.
• → The mask is the promise that fake steps contribute nothing.

The whole diagram (and this topic) is one contract: an ordered batch of vectors, plus a mask, in, and something that exploited the order, out.

Carry one analogy through the whole phase: a friend texting you a story, one word per message.

• Each message is a token. You cannot read message 14 before message 1 — the story arrives in order, and order is the story.
• After every new word you update one mental summary: who, doing what, to whom, so far. That running summary is what Topic 111 calls the hidden state.
• The word "he" in message 14 only makes sense because of message 2. That is a long-range dependency — your summary had to keep message 2 alive for 12 more messages.
• The story might be 5 messages or 500. Your brain accepts both without relearning anything — that is variable-length processing.
• Sometimes your message screen shows 4 slots but only 2 messages arrived; you glance past the empty slots. That is padding + masking.

Every architecture in Phase 6 is a different answer to one question about this scene:

RNNs answer: keep updating the summary (cheap, but old details fade). Attention answers: scroll back and re-read the whole chat log (exact, but the log grows). LSTMs and transformers are just better and worse compromises between those two strategies.

A token is the atomic step of the sequence — the unit the model consumes, one per time step. Choices at this layer often matter more than architecture tweaks:

• Character-level: the smallest alphabet (~100 symbols). No out-of-vocabulary (OOV) failures ever — any string can be spelled. But sequences become 4-7x longer and each step carries little meaning. Used for spell correction, genomics, and multilingual robustness.
• Word-level: intuitive ("one word = one step"), but the vocabulary explodes and any unseen word ("Grokified") is simply unrepresentable. Effectively abandoned for modern LLMs.
• Subword-level (BPE, WordPiece, SentencePiece): the 2024-2026 default. Llama-3 uses a 128k BPE vocabulary; GPT-4-class models tokenize at roughly 0.75 words per token for English. Rare words split into meaningful fragments ("token", "ization"), which keeps vocabularies bounded while eliminating hard OOV.
• Non-text tokens: audio becomes 25 ms frames or neural codec codes; time series becomes fixed-interval samples; user events become categorical IDs embedded into vectors.

Notice the pattern: whatever the raw stream, the tokenizer chops it into steps so the story-in-messages analogy applies unchanged.

After tokenization, every sequence is a list of integer IDs. An embedding matrix E ∈ R^{|V| × d} maps each ID to a dense d-dimensional vector — a learned lookup: token ID goes in, a row of E comes out.

A batch is then a single tensor of shape [batch, seq_len, d_model]. Example: 32 sentences of up to 128 tokens at d = 512 is one 32 × 128 × 512 float tensor.

Because GPU tensors must be rectangular, the sequences in a batch are padded to the longest one and neutralized with an attention mask so padded positions contribute nothing.

The mask is exactly what makes Transformers safe on ragged text — and the reason wasted padding bytes are a real throughput tax. Modern stacks attack that tax directly: FlashAttention varlen kernels skip the pad compute, and sequence packing (Llama-3-style training) squeezes many short documents into one full-length row.

import torch, torch.nn as nn

# vocab=50_257 (GPT-2 BPE), d_model=512, batch=32, padded length=128
ids = torch.randint(0, 50_257, (32, 128))          # token IDs   [B, T]
mask = (ids != PAD_ID)                              # 1 = real, 0 = pad

emb = nn.Embedding(50_257, 512)                    # learned E, |V| x d
x = emb(ids) * (512 ** 0.5)                         # [B, T, d] = 32x128x512
# Feed x + mask to any sequence model in this phase: RNN, LSTM, Transformer.

Sequence problems are classified by I/O cardinality — how many inputs in, how many outputs out. This vocabulary tells you which architecture family you need:

• One-to-many (image → caption; music → score): a fixed input, a generated sequence output.
• Many-to-one (sentence → sentiment; click session → fraud score): a sequence in, a single label out. Only the final state is read.
• Many-to-many, same length (POS tagging; CharRNN language modeling; sensor denoising): every input step emits an output step.
• Many-to-many, different lengths (machine translation; speech recognition; summarization): the hard case — input and output can even be in different languages. It motivated encoder-decoder architectures (Topic 116) and attention (Topic 118).

In the texting analogy: many-to-one is "read the whole story, tweet one verdict"; many-to-many is "read a story in English, text back the same story in Finnish — which may take more or fewer messages."

The design tension of the entire phase, in one sentence: how does information at step 3 influence the prediction at step 3,000?

Three families answer differently:

• RNNs (Topic 111) pass everything through one narrow bottle — a hidden state updated every step. Cheap memory, but gradients across 1,000 multiplications decay or explode (Topics 112-113).
• LSTMs/GRUs (Topics 114-115) add gates that protect the signal, extending usable memory to a few hundred steps.
• Self-attention (Topic 119) makes the distance between any two positions a single hop — at the price of quadratic compute in sequence length (Topic 120).

Every architecture in Phase 6 is a different answer to the same question: how to let distant events talk to each other without bankrupting compute or gradients.

In production, this is not academic. A chat model must still know your name from message 3 at message 400. A fraud model must link tonight's click to a pattern from three sessions ago. A forecaster must remember every Christmas. Whichever trade you pick — fading summary or growing re-read log — becomes your model's memory, its cost curve, and its failure mode.