Rotary Position Embeddings (RoPE)

Self-attention computes "how relevant is token A to token B" for every pair. Notice what that formula never mentions: where the tokens are.

Shuffle a sentence and attention sees the exact same bag of pairwise affinities. It is permutation-invariant. "dog bites man" and "man bites dog" would score identically unless you inject position information somehow.

Early solutions and their limitations, in one line each:

• Learned absolute embeddings (original Transformer): add a position-specific vector to each token. Works, but extrapolates badly past the trained length — positions it never saw are meaningless noise.
• Sinusoidal absolute embeddings: fixed sine/cosine waves per position. Also absolute — scores end up depending on where each token sits, not on how far apart two tokens are, which is what language mostly cares about.
• ALiBi: inject a distance penalty directly into attention scores. Nice relative behavior, but it caps the maximum attention span by design.

So the question becomes

RoPE (Su et al., 2021, arXiv 2104.09864; first trialed in RoFormer) takes a cleaner route: encode position by rotating the query and key vectors themselves.

RoPE in one line

The magic is a fact from plane geometry: rotating one arrow by α and another by β, then taking their dot product, gives the same answer as keeping the first still and rotating the second by (β − α). Only the relative turn matters — the two absolute headings cancel out.

Written as the identity people ask you to derive on whiteboards:

⟨ R(m·θ) q , R(n·θ) k ⟩ = ⟨ q , R((n − m)·θ) k ⟩

Read it: q at position m and k at position n are each rotated by their own absolute position — but the score they produce depends only on the relative offset (n − m). Absolute rotations in, relative attention out. That is the whole design.

The frequencies: split the d-dimensional head vector into d/2 two-dimensional slices; slice i rotates at

θᵢ = base^(−2i/d), with base = 10,000 by default

• Low-index slices (small i): fast rotation — fine local resolution, distinguishing neighbors.
• High-index slices: slow rotation — coarse, long-range signal.

And critically: RoPE is applied after the Q/K linear projections. It adds zero parameters, no extra memory, leaves V untouched, and is just an elementwise rotation — which is why it composes perfectly with fused kernels like Flash Attention.

Suppose one 2-D slice rotates 30° per token position (θᵢ = 30°).

• The query at token m = 2: its slice is turned 2 × 30° = 60°.
• The key at token n = 5: its slice is turned 5 × 30° = 150°.
• The dot product depends only on the difference: 150° − 60° = 90°.

Check the property: tokens at m = 10 and n = 13 have turns 300° and 390° — again a 90° gap, and the score contribution is identical. Same distance, same behavior, anywhere in the sequence. That is relative-position attention computed from absolute rotations.

Now the second knob — different slices, different speeds. With d = 4 (two slices) and base = 10,000:

code
Copy
 slice 0:  θ₀ = 10000^(−0/4)  = 1      → 1 radian per token: spins fast
 slice 1:  θ₁ = 10000^(−2/4)  = 0.01   → 0.01 radians per token: crawls

• After 2 tokens the fast slice has turned 2.0 rad (very different angle — great at resolving nearby order).
• After 2,000 tokens the slow slice has turned 20 rad ≈ 3 full circles (still distinguishable — great at measuring far distances).

Each frequency covers a different range, like the second, minute, and hour hands of a clock covering seconds up to twelve hours. Together they make offsets from 1 to very large uniquely identifiable.

code
Copy
  position m=2          position n=5          what the dot product sees
     ╱ (60°)                ╱ (150°)                 angle between
   ●                        ●                    arrows = 90°  →  only
                                                  n − m = 3 survives

   fast slice θ₀=1 rad/tok   slow slice θ₁=0.01 rad/tok
   hand sweeps the whole     hand needs ~600 tokens per
   dial every ~6 tokens      full turn → the far-range ruler
   → fine local print        for pairs thousands apart

Two more consequences you can see in this picture:

• Distance decay: because the many slices' angles realign less and less often as (n − m) grows, expected dot products between rotated vectors fall off with |m − n|. Attention gets an implicit locality prior for free — nearby tokens are easier to attend to unless something strongly matches.
• Absolute position is only ever an angle. Which means: out past the trained length, the hands point at angles the model literally never saw during training — and learned attention patterns break (section 6).

Carry one picture through: a watch face with dozens of hands, each ticking at its own speed.

• Each 2-D slice of the head vector is one hand. Token position is time: at position m, hand i has moved to angle m·θᵢ.
• The dot product between query and key is the watch comparing two photos of the hands. Because both photos advanced by the same rule, subtracting them erases the absolute time and leaves only the elapsed time between the tokens — relative position, extracted automatically.
• The fast hand (largest θᵢ, the low-index slices) reads minutes: it resolves whether two events were 1 or 2 apart. The slow hand (tiny θᵢ) reads days: it distinguishes events a thousand apart, which the fast hand cycles over repeatedly.
• Training length = the era your watch was built for. A watch calibrated for a 4,096-token "day" gets handed a 9,000-token day: the hands land on angles no one ever taught it to interpret, and its time-reading collapses.
• Raising the base (10,000 → 500,000) = replacing every hand with a slower one so one "day" spans a million tokens. You keep the multi-hand trick — just stretch every wavelength. The catch: slower hands are coarser, so nearby resolution blurs unless you compensate (partial compensation via mixing regimes — YaRN — or keeping some fast hands).

Everything below — PI, NTK, YaRN, LongRoPE — is a different way of re-calibrating the hands without buying a new watch (retraining from scratch).

# f(x, pos) = R(pos) x  with R the block-diagonal rotation
# For any m, n:
#   < R(m·θ) q , R(n·θ) k >  =  < q , R((n−m)·θ) k >
# i.e. the dot product sees only the RELATIVE offset (n - m).

def rope(x, pos, base=10000.0):        # x: (seq, head_dim), head_dim = 2 * n_pairs
    dim = x.shape[-1]
    i = torch.arange(dim // 2, device=x.device, dtype=torch.float)
    inv_freq = base ** (-2 * i / dim)   # θ_i
    angles = pos[:, None] * inv_freq[None, :]      # (seq, dim/2)
    cos, sin = angles.cos()[:, None], angles.sin()[:, None]
    x1, x2 = x[..., 0::2], x[..., 1::2]            # even/odd pairs
    return torch.stack([x1 * cos - x2 * sin,       # rotate each pair
                        x1 * sin + x2 * cos], -1).flatten(1)

RoPE delivers three gifts at once:

• (a) Relative semantics from an absolute formulation. Scores depend on (n − m) by construction — yet the implementation stays as cheap as absolute embeddings and compatible with Flash-style fused kernels, since it is elementwise on q/k.
• (b) Distance decay. The natural preference for nearby tokens (section 4) is a useful prior for language, achieved without a single added parameter.
• (c) No parameters, no extra memory. Nothing to learn, nothing to store.

By 2023 it had conquered open-weights: Llama 1/2/3, Mistral/Mixtral, Qwen, DeepSeek (with partial rotary dimensions in MLA), Gemma, Phi-3+ — essentially every major family uses RoPE.

Its weakness is the flip side of the design: trained models only ever saw rotations up to their training length L. At positions beyond L, high-frequency slices produce out-of-distribution angles — hands pointing at never-seen markings — and attention collapses. That is the infamous long-context breakdown.

Also, a single global base ties together all the wavelengths: you cannot sharpen nearby resolution without compressing long-range resolution, and vice versa. The fix families:

The 2024-2026 practice simplified the story: instead of interpolating at deploy time, frontier pretrains pick a much larger base and train long from scratch — build a watch whose slowest hand spans the target era from day one.

• Llama 3 moved base 10,000 → 500,000 for its 128K context.
• 1M-context configurations (Gemini 1.5-class, Llama 4, Kimi) push bases into the millions, stretching the slowest wavelength to comfortably cover million-token spans.

This is why "RoPE base" is now a visible config knob in HF model cards (e.g., rope_theta: 500000).

Two more 2024-2026 refinements:

• Partial rotary dimensions: DeepSeek MLA applies RoPE to only 32 of 192 dims, keeping the rest position-free so the bulk of the KV cache can be compressed independently of positional signal.
• Per-layer frequency allocation: different layers get different hand-speed mixes, mitigating "attention dilution" at extreme distances.

Research into alternatives continues (HoPE, NoPE — training without positional embeddings can surprisingly extrapolate on retrieval tasks), but RoPE plus base scaling plus YaRN-style training remains the industry workhorse into 2026.