Vector and Matrix Norms: L1, L2, and Measuring Magnitude

You have a weight vector w = (3, −4).

Someone asks: how big is it?

That sounds like one question with one answer. It doesn't.

• Count raw magnitude, ignoring signs: 3 + 4 = 7.
• Pythagoras: √(9 + 16) = 5.
• "How big is the worst single entry?" → 4.

All three are legitimate. All three give a different number.

And in machine learning, the number you choose is not cosmetic. It decides:

• what your loss punishes (a big mistake once, or small mistakes everywhere?),
• whether useless weights die exactly zero or just get small,
• whether one exploding gradient can wreck a training step — norms are how clipping catches it.

So before losses, regularization, and gradient hygiene make sense, you need norms. That's this topic.

A norm ‖x‖ is simply

To deserve the word "size", it must follow three common-sense rules:

1. ‖x‖ = 0 only for the zero vector x = 0 — nothing has zero size except nothing.
2. ‖αx‖ = |α|·‖x‖ — double the vector, double its size.
3. Triangle inequality: ‖x + y‖ ≤ ‖x‖ + ‖y‖ — going directly is never longer than detouring. (Walk one leg, then the other: total ≥ straight path.)

Different formulas satisfy those rules in different ways. The family you must know:

• L1 norm ‖x‖₁ = Σ|xᵢ| — "Manhattan" length; sum of absolute components.
• L2 norm ‖x‖₂ = sqrt(Σ xᵢ²) — Euclidean length; ‖x‖₂² = x·x (Topic 3: a vector dotted with itself sums its squares).
• L∞ norm ‖x‖∞ = max|xᵢ| — just the largest component (used for clipping, worst-case bounds).
• Squared L2 Σ xᵢ² — not a true norm but smoother and cheaper to optimize; the backbone of MSE.
• L0 "norm" (count of nonzeros) — not a true norm either, but it is what "sparsity" talk really wants.
• Frobenius norm for matrices: ‖A‖F = sqrt(Σᵢⱼ aᵢⱼ²) — flatten the matrix into a long vector and take its L2; also equals the sum of squared singular values.

Subtract two vectors first and the same formulas become distances: d(x, y) = ‖x − y‖₁ (Manhattan), ‖x − y‖₂ (Euclidean), ‖x − y‖∞ (Chebyshev — the biggest coordinate gap).

Take x = (3, −4) and run every ruler over it.

L1: add the absolute values.

‖x‖₁ = |3| + |−4| = 7

L2: Pythagoras.

‖x‖₂ = √(3² + 4²) = √25 = 5

(The famous 3-4-5 right triangle: the vector is the hypotenuse.)

L∞: take the biggest magnitude.

‖x‖∞ = max(3, 4) = 4

Squared L2: skip the root.

Σxᵢ² = 9 + 16 = 25

One vector, four honest sizes: 7, 5, 4, 25. Each answers a different question — "total amount?", "straight-line reach?", "worst entry?", "smooth penalty?"

Now measure a distance, to y = (1, 2). The gap is x − y = (2, −6):

• Manhattan: |2| + |−6| = 8
• Euclidean: √(4 + 36) = √40 ≈ 6.32
• Chebyshev: max(2, 6) = 6

The ranking flips with the shape of the gap — which is exactly why the choice of norm matters for losses and retrieval. The code below does all of this in NumPy.

import numpy as np

x = np.array([3.0, -4.0])
print(np.linalg.norm(x, ord=1))    # 7.0   L1
print(np.linalg.norm(x, ord=2))    # 5.0   L2 (the 3-4-5 vector)
print(np.linalg.norm(x, ord=np.inf))  # 4.0 L-inf
print(x @ x)                       # 25.0  squared L2 via dot product

W = np.random.randn(3, 4)
print(np.linalg.norm(W, 'fro'))    # Frobenius norm of a matrix

y = np.array([1.0, 2.0])
print(np.abs(x - y).sum(), np.linalg.norm(x - y))  # L1 and L2 distances

The fastest way to feel a norm: collect every vector whose norm equals 1 and color them in. Each norm paints its own "unit ball":

code
Copy
     L1: diamond          L2: circle          L∞: square

          ▲                     ▲                     ▲
         ╱ ● ╲                ╭─┼─╮                ┌──●──┐
        ●  ●  ●              ●  ●  ●              ●  ●  ●
         ╲ ● ╱                ╰─┼─╯                └──●──┘
        ╱     ╲                 │                  │     │
   ────────►   │            ────┴────►         ────┴─────┴──►
   corners sit ON            perfectly          flat sides,
   the axes                  round              no corners

• L1's ball is a diamond with sharp corners on the coordinate axes.
• L2's ball is a circle — perfectly round, no corners anywhere.
• L∞'s ball is a square aligned with the axes.

Hold two facts in your head and never re-derive them:

1. The diamond's corners are places where one coordinate is exactly zero.
2. The circle's boundary never touches an axis — points near an axis still have every coordinate small but nonzero.

Section 7 shows these two facts become "L1 = feature selection" and "L2 = gentle shrinkage". This is the single most-asked norms interview question, answered by geometry.

Here is the analogy to carry through everything below:

Training wants big weights; a budget forces you inside a fence.

Picture the loss as a hilly terrain whose low points pull the weight vector w outward, growing it. Regularization says: fine, but you must live inside a fence of size λ (λ‖w‖ ≤ budget). You slide down the loss and press against the fence; training stops where the two balance — where the loss contour first touches the fence.

Now the punchline: the shape of the fence decides where you get pinned.

• A diamond fence (L1) has pointy corners sticking out along the axes. The touch point lands on a corner surprisingly often → one weight takes the whole budget, the rest sit exactly at zero. The fence itself deleted features for you.
• A round fence (L2) has no corners. The touch point lands mid-edge → every weight survives, all shrunk a little. Nothing is deleted; everything is softened.
• A square fence (L∞) has flat sides: it only ever pins the single largest coordinate and lets the others ride freely — that's the logic of clipping.

Same loss, same budget. Only the shape of the ruler changed. Keep this picture; Sections 6–8 are all "which fence?" questions.

Every regression loss is "penalize the error e = prediction − target with some norm." The choice has teeth:

• MSE (squared L2 error): smooth, differentiable everywhere, and the maximum-likelihood loss under Gaussian noise. But squaring punishes outliers brutally — one 10× error costs 100× — so it is sensitive to heavy-tailed noise and mislabeled data.
• MAE (L1 error): robust to outliers (a 10× error costs 10×); corresponds to Laplace noise. Its gradient is constant-magnitude (just the sign of the error), so convergence slows near the optimum and plain subgradients are coarse.
• Huber loss: the fence that's round near home and straight far away — L2 for small residuals, L1 for large. The best of both, standard in RL (e.g. PPO/DQN value heads) and robust regression.
• KL divergence / cross-entropy generalize "distance" to distributions; classification trains on them, not on raw norms.

Sanity-check the outlier claim with numbers: errors of 1 and 10 cost MSE 1 + 100 = 101 but MAE 1 + 10 = 11. One corrupted label can literally steer the whole batch gradient under MSE.

Norms also appear outside the loss, as direct constraints and meters:

• gradient clipping by L2 norm (torch.nn.utils.clip_grad_norm_) — if ‖g‖₂ exceeds the threshold, rescale the step;
• weight-norm monitoring to detect exploding gradients;
• nearest-neighbor search in vector databases: L2 vs cosine vs inner-product indexes (Topic 3) are different fences around your embeddings.

Add a norm penalty to the loss and you bias learning toward "small" parameters — the literal fence of Section 5:

• Ridge (L2, "weight decay"): + λ‖w‖₂². Keeps all weights, shrinks them smoothly, and improves conditioning of the problem (recall Topic 6). It has the cleanest optimization/implicit-bias story for deep nets. Important nuance: AdamW (2019 → default through 2026) decouples decay from the adaptive gradient scale — the update is w ← w − lr·(update + λw) — which is the true L2 penalty practitioners mean when they say "weight decay".
• Lasso (L1): + λ‖w‖₁. Because the L1 diamond's corners sit on the axes, optima frequently land exactly on one → coefficients become truly zero: built-in feature selection, sparse models, interpretability for tabular data.
• Elastic net: mix both penalties — L2 stabilizes L1 when features are correlated and the diamond can't decide between twins.

The geometric argument is the standard explanation from the elements (the "diamond vs ball" picture in the ESL book, §3.4, drawn above in the unit-ball section).

For whole matrices, beyond Frobenius there is the spectral norm ‖A‖₂→₂ = σmax(A) (the largest singular value, Topic 6): the maximum stretch the matrix can apply to any unit vector. It underpins a lot of modern practice:

• Condition number κ = σmax/σmin — how much solving linear systems amplifies error (Topic 6); normalization layers (LayerNorm) tame it in networks.
• Lipschitz-constrained nets: bounding layer spectral norms stabilizes GANs and gives robustness/generalization certificates — a fence again, this time around the stretching, not the weights.
• Spectral clipping / μP scaling: modern LLM training monitors and clips gradient matrices by spectral properties (e.g. the Muon optimizer, 2024–2026, orthogonalizes momentum matrices via Newton–Schulz iteration).
• Low-rank pressure: Frobenius distance ‖W − W'‖F is exactly what LoRA-style adaptation and model-merging evaluate — merging is "stay within ε in Frobenius norm of a good model."