Positional Encoding: Injecting Order into Attention

Here is the embarrassing fact about the layer you just learned (Topic 119):

• "dog bites man"
• "man bites dog"

To self-attention, these are the same input — the same three vectors, the same all-pairs similarities, just listed in a different order.

Exactly: attention scores pairwise content similarity. It never asks "who came first?"

Prove it in one line: for any permutation matrix P, self-attention satisfies

Attn(P·X) = P · Attn(X) — permutation equivariant.

Shuffle the tokens and the outputs shuffle identically; the layer has no mechanism to notice.

RNNs (Topic 112, in one sentence: they consume tokens one at a time) got order for free from the loop — first token first, always.

Transformers threw the loop away for parallelism — and with it, the free order. They must be told where each token sits, by adding a position-dependent signal to (or shaping it inside) the vectors themselves.

Requirements any good encoding must satisfy:

• Distinguish position 3 from position 70 in training range.
• Support relative reasoning: what matters linguistically is "3 back," not "index 214."
• Ideally extrapolate: handle T_test > T_train without collapse (long-context economics, Topic 122's cousin).

Three wishes. The 2017-2026 history is four candidate solutions trading them off.