Eigenvalues and Eigenvectors: The Fixed Axes of a Transformation

Imagine a robot that drags every arrow drawn on the floor.

You feed it an arrow. It spits out a new arrow — usually pointing somewhere else, with a different length.

That robot is a matrix (a rectangular grid of numbers that turns one vector into another).

Most arrows get knocked off their line:

• An arrow pointing east comes out pointing northeast.
• An arrow pointing north comes out pointing east.

Rotated. Hard to think about.

So the question becomes

There almost always are. A few lucky directions come out parallel to how they went in — only their length changed.

Those lucky directions are the eigenvectors, and the stretch factors are the eigenvalues.

Why should you care? Because "which directions does this transformation truly stretch?" hides inside three very different ML problems:

• PCA: which axes is my data longest along? → shrink the data without losing much.
• Loss landscapes: which direction curves sharpest? → that caps your learning rate.
• Deep/recurrent networks: does a signal grow (explode ↑) or shrink (vanish ↓) when it is multiplied layer after layer?

One idea answers all three.

An eigenvector is simply

Formally: for a square matrix A, a nonzero vector v is an eigenvector with eigenvalue λ if:

A v = λ v

Say the equation out loud: "run v through A, and you get v back, times a number λ." The left side is the machine's work; the right side is the proof the direction survived.

Now unpack what the eigenvalue is telling you:

• λ > 1: that direction is amplified.
• 0 < λ < 1: damped — shorter, but same ray.
• λ < 0: the arrow flips to the opposite ray.
• λ = 0: the direction is crushed to nothing — the matrix is singular (Topic 6).

How do you find them? Two steps:

1. Solve det(A − λI) = 0 for λ. (Topic 6: a determinant of zero means the matrix collapses space — here it means some stretch factor makes a whole direction vanish.)
2. For each λ, back-substitute into (A − λI)v = 0 to get that direction v.

The friendly case is symmetric (or Hermitian) matrices. They are the ones ML actually uses, and they behave beautifully:

• All eigenvalues are real (no imaginary surprises).
• Eigenvectors come in a complete set of orthogonal directions.
• Everything packs into A = QΛQᵀ, the spectral decomposition: rotate into eigen-coordinates, stretch along each axis, rotate back.

Covariance matrices and Hessian matrices are symmetric — which is exactly why eigendecomposition shows up everywhere in machine learning.

Take the symmetric matrix

A = [[2, 1], [1, 2]]

Step 1 — find the eigenvalues. det(A − λI) = (2−λ)² − 1 = 0, so 2−λ = ±1:

λ₁ = 3, λ₂ = 1

Step 2 — find the directions. Back-substitute (A − λI)v = 0:

• For λ₁ = 3: the equations force v along (1, 1). Check by hand: A·(1,1) = (2+1, 1+2) = (3, 3) = 3·(1,1). ✓ Same line, 3× longer.
• For λ₂ = 1: v is forced along (1, −1). Check: A·(1,−1) = (2−1, 1−2) = (1, −1) = 1·(1,−1). ✓ Same line, exactly as long. Notice the two directions are perpendicular — the symmetric-matrix bonus.

Any other arrow fails. Try v = (1, 0): out comes (2, 1) — off its line. Not an eigenvector.

Step 3 — the payoff: power iteration. Multiply a random vector by A again and again, renormalizing each round:

(1,0) → (2,1) → (5,4) → (14,13) → (41,39) → …

The x/y ratio slides toward 1 — the direction keeps crawling onto the λ = 3 line. The biggest eigenvalue always wins repeated multiplication. That loop is exactly what the last lines of the code below do, and it is how PageRank works at web scale.

import numpy as np

A = np.array([[2.0, 1.0], [1.0, 2.0]])   # symmetric
vals, vecs = np.linalg.eig(A)            # eigh() for symmetric matrices
print(vals)                              # [3. 1.]
print(vecs)                              # columns are eigenvectors

v = vecs[:, 0]
print(A @ v, vals[0] * v)                # identical: A v = lambda v

# Repeated application amplifies the dominant eigen-direction
x = np.random.randn(2)
for _ in range(20):
    x = A @ x / np.linalg.norm(x)        # power iteration -> v1
print(np.sign(x) @ np.sign(vecs[:, 0]))  # ~1: aligned with eigenvector 1

Draw the three arrows from the example before and after the machine A runs:

code
Copy
   before A              after A             verdict
   ----->  (1, 0)        ------>↗ (2, 1)     knocked off its line ✗
   ----->↘ (1,−1)        ----->↘ (1,−1)      same line, ×1 stretch ✓
   ↗                                  ↗↗↗
   ↗      (1, 1)                      ↗       same line, ×3 stretch ✓
   ↗                                  ↗

Why do only the diagonals survive? Because a symmetric matrix stretches space by different amounts along two perpendicular grain directions — here (1,1) gets ×3 and (1,−1) gets ×1.

Any other arrow is a blend of the two grains. For instance:

(1, 0) = ½·(1, 1) + ½·(1, −1)

Apply A: the ½·(1,1) piece triples into 1.5·(1,1); the ½·(1,−1) piece stays. The blend's recipe changed, so its direction bent.

Only pure-grain arrows come out parallel. That is the whole geometry of "Av = λv".

And the eigen-directions of a quadratic function (like a loss near a point) are exactly these stretch axes: the inverse of the stretch factor is how wide the valley is in that direction. Keep that picture — Section 7 uses it.

Carry one analogy through the rest of the topic: a block of wood you can stretch.

Wood has a grain. How you pull decides what happens:

• Pull along the grain → the block just gets longer. Clean, predictable. That pull is an eigenvector, and "how many times longer" is the eigenvalue.
• Pull across the grain → the block bends sideways. That is your rotated, knocked-off-line arrow.
• The grain exists because of the block, not because of how you pull. Every matrix hides its own grain; eigendecomposition is simply finding it.
• λ = 3: grain triples. λ = 0.5: grain halves. λ = −1: it shoots out backwards. λ = 0: that grain snaps flat — information destroyed.

Power iteration, then, is just yanking the block over and over. Each yank favors the strongest grain, so after a few rounds the shape screams "λ₁ direction." Cheap to run — one matrix multiply per yank.

Everything else in this topic is reading the grain of whatever matrix you care about:

• grain of the data → PCA
• grain of the curvature → learning-rate limits
• grain of the connectivity → PageRank and graph nets

Principal Component Analysis is eigendecomposition applied to the data covariance matrix Σ = XᵀX / n (sklearn actually uses SVD, which generalizes it — PCA's components are the eigenvectors of Σ, and each component's explained variance is its eigenvalue).

Why is the grain of Σ interesting? Because Σ encodes how your data spreads in every direction at once. Its eigenvectors are the data's natural axes:

• Top eigenvector = direction of maximum variance — the grain the data is longest along.
• Second eigenvector = the most variance remaining orthogonal to the first; and so on down the list.
• Compress to k dimensions by projecting onto the top-k eigenvectors. This is provably the best k-dimensional summary in the least-squares sense (Eckart–Young theorem). "Keep 95% of explained variance" means: keep the axes whose eigenvalues add up to 95% of the total.

2024–2026 relevance: PCA-whitening still preprocesses embeddings, top_p-style spectral analyses examine activation covariance, and modern intrinsic-dimension estimators — used to argue LLMs are far smaller than their parameter counts — rest on eigenvalue spectra of layer weights and gradients.

Now let the matrix be the Hessian H of the loss — the matrix of second derivatives that measures how sharply the loss surface bends (Topic 13). H is symmetric, so it eigen-decomposes as H = QΛQᵀ, and each eigenvalue is the curvature along its eigenvector direction:

• Large λ (sharp canyon): a learning rate above ≈ 2/λ diverges in that direction; optimizers must adapt (Adam, and K-FAC / Shampoo / Sophia approximate H⁻¹ — Topic 6).
• λ ≈ 0 (flat basin): little signal; progress along that direction is slow.
• λ < 0: you are near a saddle or maximum — SGD noise famously escapes saddles but not minima.
• Ill-conditioned spectrum (the ratio λmax/λmin huge): explains why gradient descent zig-zags on naive quadratic losses; whitening/normalization layers try to flatten the spectrum.

In the wood analogy: λ is the stiffness of each grain, and the learning rate is how hard you yank. Yank harder than the stiffest grain can take and that one direction blows up — and it drags the whole training run with it.

The same readout decides what happens through depth. For recurrent nets and depth-stacked Jacobians, whether activations grow or die is decided by eigenvalues (more precisely singular values) relative to 1:

• |λ| > 1 → signal explodes ↑ layer after layer
• |λ| < 1 → signal vanishes ↓ toward zero

That is the quantitative form of exploding/vanishing gradients (Topic 12).

The "eigenvectors of some matrix" pattern keeps reappearing:

• PageRank / graph neural nets: dominant eigenvector of the (damped, possibly normalized D^(−1/2) A D^(−1/2)) adjacency matrix; spectral graph convolutions filter along its eigenvectors.
• Spectral clustering: eigenvectors of the Laplacian L = D − W reveal cluster structure.
• Attention/similarity graphs: clustering embeddings via leading eigenvectors of a similarity matrix (affinity-propagation neighbors).
• Diffusion / sampling: the Fokker–Planck and noise-schedule analyses read eigenvalues of the generator's transition operator.
• Spectral regularization & sharpness-aware training: penalize the top Hessian eigenvalues to prefer flat minima that generalize better — an active research line through 2024–2026.

Note the economy: none of these needs the full decomposition. Power iteration — yank, normalize, repeat — grabs just the dominant eigenpair at the price of one matrix-vector product per round. That is why Google can rank billions of web edges without ever eigensolving a dense matrix.