Identity and Inverse Matrices: Doing Nothing and Undoing Anything

From Topic 4 you know a matrix is a transformation: it stretches, rotates, and shears space, and y = Ax runs your input x through it.

So a natural question appears

Sometimes yes, sometimes no. A rotation you can undo by rotating the other way. But a machine that flattens everything onto a line throws information away — you can't recover what it erased.

Answering this gives you two characters you'll meet in every corner of ML: the identity (the do-nothing machine) and the inverse (the undoing machine). And, crucially, it explains why real training code almost never asks for an inverse directly.

Start with the do-nothing machine.

It's the n×n grid with 1s on the diagonal and 0s elsewhere. It's the multiplicative do-nothing element: AI = IA = A — just like multiplying a number by 1.

Now the undoing machine.

You already meet the identity constantly in ML, without seeing the letter I:

• Residual connections y = x + f(x) = (I + F) applied to x — the skip path is literally an identity. (Topic 2.)
• BatchNorm/LayerNorm scale initializes as γ = 1 so the layer starts as (near) identity behavior.
• Adam-family optimizers and decoupled weight decay add/subtract multiples of the weight vector — an I-shaped operation on parameters.
• Initialization theory: you want the product of layer Jacobians to behave like an identity (preserve signal variance) — the motivation behind Xavier/He initialization.

An undoable one. Let A = [[2, 0], [0, 3]] — "double the x-coordinate, triple the y-coordinate." Feed in x = [1, 1]:

• Ax = [2, 3]
• To get back, divide each coordinate by the number it was scaled by: the undoing machine is A⁻¹ = [[1/2, 0], [0, 1/3]].
• Check: A⁻¹(Ax) = [1, 1] = x. It came home, so A⁻¹A = I. No direction was erased, so every direction can be reversed.

An un-undoable one. Let S = [[1, 2], [2, 4]]. Notice the second row is just 2× the first — the two inputs get mixed into a single line of output.

• Feed [2, -1]: S·[2,-1] = [0, 0]. A nonzero vector got squashed to zero.
• Feed [0, 0]: also [0, 0].
• Now you're handed output [0,0] and asked "what came in?" — impossible; infinitely many answers, information is gone.

So S has no inverse. It's singular. That single example is the whole difference between the two matrices in the diagram: one preserves everything (recoverable), one squashes a direction to zero (information lost forever).

import numpy as np

A = np.array([[4.0, 7.0], [2.0, 6.0]])
Ainv = np.linalg.inv(A)
print(A @ Ainv)                 # ~[[1, 0], [0, 1]]

S = np.array([[1.0, 2.0], [2.0, 4.0]])   # rank 1: second row = 2x first
try:
    np.linalg.inv(S)                       # LinAlgError: singular
except np.linalg.LinAlgError:
    print("no inverse")

print(np.linalg.pinv(S))        # pseudoinverse still works (least squares)
print(np.linalg.cond(A))        # ~ condition number: how safe is solving?

For a square matrix, all these say the same thing — memorize them as one bundle:

• det(A) ≠ 0
• Full rank: columns are linearly independent; the map is a bijection (no dimension squashed).
• Ax = 0 has only the trivial solution.
• No zero eigenvalue (Topic 7) — every direction is stretched by a nonzero amount.

When a matrix isn't square (or is singular), the Moore–Penrose pseudoinverse A⁺ steps in: x = A⁺b gives the minimum-norm least-squares solution — the exact math behind ordinary least squares regression. It's the "best you can do when a true undo is impossible."

Carry one image through the rest: a machine that reshapes a lump of clay.

• The identity is the pass-through slot that leaves the lump exactly as it was. Boring, but it's the reference point for "nothing changed."
• An invertible machine stretches and tilts the lump without crushing any dimension. A second machine (the inverse) runs the lump back and restores the original shape perfectly — because nothing was ever lost.
• A singular machine flattens the lump into a pancake, erasing the height dimension completely. No reverse machine can rebuild a lost dimension — so no inverse exists.
• A nearly-singular machine leaves the lump very thin in one direction. It's technically reversible, but the tiniest fingerprint smudge on the pancake explodes into a huge error when you try to un-flatten it.

That last bullet is the real-world danger, and the code block's np.linalg.cond measures exactly how thin the thinnest direction is.

To fit the classic normal equations for linear regression you would compute β = (XᵀX)⁻¹Xᵀy. In production you almost never form the inverse explicitly:

• Numerical stability: solvers like numpy.linalg.solve, Cholesky/QR factorizations, and conjugate gradient avoid the error amplification of an explicit inverse (and are cheaper: solving one system costs about a factorization, while inv costs factorization × n).
• With many right-hand sides or huge systems (millions of features), direct inversion is memory- and FLOP-prohibitive; gradient-based optimization or iterative solvers take over.
• If XᵀX is singular (collinear features, p > n), there is no inverse at all — yet training still must move.

The fix is the single most important I in machine learning: ridge regularization solves (XᵀX + λI)β = Xᵀy. Adding λ·I shifts every eigenvalue up by λ, guaranteeing invertibility, taming the condition number, and trading a little bias for huge stability. In the clay analogy, λ·I is a shim that stops any direction from being pressed all the way to zero. Lasso, weight decay, and Adam's epsilon denominator are cousins of this same idea.

Modern systems approximate inverses rather than compute them:

• Normalizing flows define generative models via invertible transformations whose Jacobian determinant is tractable (RealNVP, affine coupling).
• Newton methods (e.g. K-FAC, and 2024–2026 second-order optimizers like Sophia and Shampoo) use curvature/Hessian approximations in place of true matrix inverses — the inverse of the second derivative from Topic 9, generalized.
• Whitening in vision pipelines multiplies data by an inverse square root of the covariance, typically via eigendecomposition with an epsilon floor — again avoiding naive inversion.