Transpose of a Matrix: Flipping Axes, Views, and Backprop

You already know (Topic 4) that matmul only works when the inner shapes match: (m×k) times (k×n).

Now imagine attention in a Transformer. Queries come in as a (T, d) table — T rows, d columns. Keys come in the same shape. You want a score for every query against every key, which means you want (T, d) @ (d, T).

But the keys arrive as (T, d), not (d, T). The inner dimensions don't line up.

So the question becomes

That flip is the transpose. And the surprising bonus: in real code, doing it costs almost nothing.

The definition in one line

Formally, for an m×n matrix A, the transpose Aᵀ is the n×m matrix with (Aᵀ)ᵢⱼ = Aⱼᵢ. Rows become columns; columns become rows.

Some algebra identities you'll use constantly:

• (AB)ᵀ = BᵀAᵀ — the order reverses (important when deriving gradients).
• (Aᵀ)ᵀ = A — flip twice, you're back home.
• (αA + βB)ᵀ = αAᵀ + βBᵀ — transpose is itself a linear operation (from Topic 2).
• A is symmetric if Aᵀ = A (e.g. covariance matrices, attention's theoretical limit cases); skew-symmetric if Aᵀ = −A.

For vectors, transposing turns a column into a row — which is why a · b = aᵀb: the dot product (Topic 3) is really a (1×n)(n×1) matmul that spits out a single number.

Start with A = [[1, 2, 3], [4, 5, 6]] — shape (2, 3): 2 rows, 3 columns.

Read the rule (Aᵀ)ᵢⱼ = Aⱼᵢ literally: pull the entry at (i, j) from the swapped position.

• Aᵀ has shape (3, 2).
• Row 0 of Aᵀ = column 0 of A = [1, 4]
• Row 1 of Aᵀ = column 1 of A = [2, 5]
• Row 2 of Aᵀ = column 2 of A = [3, 6]

So Aᵀ = [[1, 4], [2, 5], [3, 6]].

Now feel the payoff: A was (2, 3). To do A @ B you need B with 3 rows. If B happens to be (2, 3) too — wrong shape — use Bᵀ, which is (3, 2), and A @ Bᵀ works because inner 3 matches. Half of every "matmul shape mismatch" you'll ever debug is just this: insert a transpose to make the inner dims agree.

Picture the matrix written on paper, and fold it along its top-left to bottom-right diagonal. The diagonal entries stay put; everything above the diagonal swaps places with its mirror image below.

code
Copy
   A  (2 rows, 3 cols)        fold ↘ on the diagonal       Aᵀ (3 rows, 2 cols)
  ┌──────────┐                                                    ┌──────┐
  │ 1  2  3 │  ← row 0                                            │ 1  4 │
  │ 4  5  6 │  ← row 1                                            │ 2  5 │
  └──────────┘                                                    │ 3  6 │
       diagonal: 1, 5                                             └──────┘

• Diagonal (1, 5) doesn't move. Every off-diagonal pair swaps with its mirror: the 2 at (0,1) trades places with the 4 at (1,0), and 3 at (0,2) trades with the entry below it — always (i, j) ↔ (j, i).
• Flip twice → back to the start, because (Aᵀ)ᵀ = A.

The same picture works for higher-rank tensors, just "fold" between any two axes instead of two rows/columns — that's the X.transpose(0, 2, 1) line in the code above, swapping the last two axes of a (heads, seq, dim) batch.

Carry one image through the rest: a classroom seating chart.

The students sit in fixed chairs. A teacher can read the room two ways:

• Row-by-row ("front row, left to right, then next row…").
• Column-by-column ("aisle 1 from the door, then aisle 2…").

Transposing is exactly that second reading. No student moves. You only changed the address scheme from "(row, column)" to "(column, row)."

That's why transposing is nearly free in code — you don't re-seat anybody, you just relabel. But notice the cost in the analogy too: if the teacher now walks up and down aisles, they zig-zag across the room instead of strolling in tidy lines. Reading order changed, so how smoothly you can sweep the room changed. That's the memory-locality catch, next section.

Arrays live in memory as a flat buffer plus shape and strides — how many elements to skip to advance one index along each axis. Transposing just swaps the stride tuple, so NumPy and PyTorch return a view in O(1): no allocation, no data movement. (In the analogy: same chairs, new label rule.)

The catch: a transposed view has non-contiguous memory — the "zig-zag across aisles" problem — so the next element-wise kernel may be slower, and some APIs (certain BLAS calls, ONNX exporters, JIT compilers) will implicitly contiguous()/copy. Rule of thumb:

• Metadata transpose (view): free, do it constantly.
• Materialized copy (.copy(), ascontiguousarray): pays O(m·n) once (re-seat everyone), then subsequent access is fast again.
• cuBLAS hides even the copy: GEMM supports op(A) = Aᵀ flags, reading the same buffer with transposed indexing — this is why F.linear(x, W) computing x @ Wᵀ costs nothing extra in PyTorch.

Once you can see the swap, you'll find it everywhere:

1. Attention (Topic 3): scores = Q @ Kᵀ. Queries are (…, T, d); you need keys as (…, d, T) so inner dimensions align — hence the transpose. Transformer code also transposes between (batch, seq, heads·dim) and (batch, heads, seq, dim) layouts.
2. Backprop through a linear layer: with Y = XW, gradients flow as dX = dY·Wᵀ and dW = Xᵀ·dY. The transpose reverses which space the gradient lives in — matrix calculus says gradients travel "against" the forward shape (see Topic 12).
3. Data-layout conversions: NCHW ↔ NHWC (channels-first vs channels-last) in training pipelines; permute in vision code.
4. Feature/label alignment: sklearn expects X as (samples, features); a transposed design matrix from stats textbooks (features × samples) must be flipped.
5. Gram/covariance matrices: XᵀX (features × features covariance) and XXᵀ (sample-sample similarity) are the two classic products, and choosing between them is the primal vs kernel trick (dual formulations predate and underlie SVMs).

Beyond plumbing, transpose connects deep ideas: AᵀA is always symmetric positive semi-definite (the normal equations come from it); the columns of A become the rows of Aᵀ, so row space ↔ column space swap; and an orthonormal matrix satisfies QᵀQ = I — meaning its inverse is its transpose, which PCA and rotation-based methods exploit constantly (Topics 6–7).