Multi-Layer Perceptrons: Depth, Composition, and Universal Approximation

Topic 75 in one plain sentence: a perceptron multiplies inputs by weights, adds a bias, and answers yes/no — so it can only ever draw one straight line through its inputs.

That line solves spam, pass/fail, fraud.

But remember the disaster at the end of topic 75:

A single perceptron fails XOR no matter how long you train it, because the right weights simply do not exist.

So the question becomes

That team is the Multi-Layer Perceptron (MLP) — also called a fully-connected or dense network.

It is the architecture that:

• broke the XOR ceiling in 1986,
• became the backbone of classical deep learning, and
• still hides inside every single layer of every transformer in every LLM you have ever used.

Understanding the MLP is understanding the "thinking with stacked layers" idea that the entire field runs on.

An MLP is simply

Written as math:

code
Copy
h1 = φ1(W1·x + b1)      # hidden layer 1
h2 = φ2(W2·h1 + b2)      # hidden layer 2
ŷ  = W3·h2 + b3          # output layer

Unpack the symbols:

• W1, W2, W3 are weight matrices — just many perceptrons' weights stored side by side. Each column is one unit's row of the score sheet.
• b1, b2, b3 are the bias vectors — one grumpiness number per unit.
• φ (phi) is a nonlinear activation — a tiny per-number twist like ReLU's max(0, z).
• h1, h2 are the hidden layers: units that never talk to the outside world. They exist purely to invent new coordinates for the layers above.

The crucial trick, in one sentence:

The final layer is then allowed to be a plain linear readout... but only because the hidden layers moved the data into a friendlier space.

XOR's four inputs: (0,0) → 0, (1,1) → 0, (1,0) → 1, (0,1) → 1. One line cannot split them. Let's do it with two hidden ReLU units and one output unit — real numbers, no hand-waving.

Hidden unit 1 asks: "is the total above 1?" → h1 = max(0, x1 + x2 − 1) Hidden unit 2 asks: "is the total below 2?" → h2 = max(0, 2 − x1 − x2)

Compute both for all four corners:

code
Copy
(x1,x2)   h1 = max(0, x1+x2−1)   h2 = max(0, 2−x1−x2)   (h1,h2)
(0,0)     max(0, −1) = 0         max(0, 2)  = 2          (0, 2)
(1,1)     max(0,  1) = 1         max(0,  0) = 0          (1, 0)
(1,0)     max(0,  0) = 0         max(0,  1) = 1          (0, 1)
(0,1)     max(0,  0) = 0         max(0,  1) = 1          (0, 1)

Look what just happened. The two negative corners became (0,2) and (1,0). The two positive corners became the same point (0,1).

In the original map, the classes were diagonal and inseparable. In the new (h1,h2) map, the negatives both have h1 + h2 = 2 while the positives have h1 + h2 = 1.

So one line in the new space finishes the job:

code
Copy
ŷ = 1  if  −h1 − h2 + 1.5 ≥ 0   (output weights (−1, −1), bias 1.5)
check: negatives → −2 + 1.5 = −0.5 → ŷ = 0 ✓
check: positives → −1 + 1.5 = +0.5 → ŷ = 1 ✓

The output layer is a plain linear perceptron — just trained on the coordinates the hidden layer invented.

That is the whole recipe: the hidden layer bends the space; the last layer cuts straight.

Picture the input plane as a sheet of paper with colored dots on it — XOR's positives red, negatives blue, one color on each diagonal.

code
Copy
 BEFORE (input space)          AFTER (hidden layer's space)

   ⊗ · · · · ○                  ○ = positive
   ·   ╲  ╱  ·        fold,     ● ● ● ● ●  ← a single horizontal
   ·     ╳    ·     pull,      · · · · · ·    SCRAPE now separates
   ·   ╱  ╲  ·     crease...    ●  NEGATIVES   everything
   ○ · · · · ⊗                  (all pushed up)
   no line works                positives collapsed together —
                                one straight cut works

Each hidden unit is one crease: w·x + b = 0 is a fold line, and ReLU decides which side of the crease stays flat and which side gets pressed to zero.

A layer of k units makes k creases at once. Stack layers and you fold the paper again on top of itself, creasing the creases.

Deeper means more folds per square inch — and the final linear readout is just one scissor-snip through the folded stack.

The rigorous version of this picture: geometrically, a hidden layer carves input space into polytopes — each ReLU unit contributes a half-space (one side of a fold line), and the network traces paths through intersections of them.

With enough hidden units, an MLP partitions the input space into an arbitrary number of convex polytopes and assigns each a (piecewise-linear) output value.

And depth is not cosmetic: deeper networks can produce an exponential number of linear regions relative to width in some architectures — Montufar et al. (2014) showed piecewise-linear networks with depth d and fixed width can realize exponentially more regions than shallow ones.

Here is the one algebraic fact the entire field of deep learning stands on.

Feed an input through two layers with no activation between them:

code
Copy
ŷ = W2·(W1·x + b1) + b2
  = (W2·W1)·x + (W2·b1 + b2)
  = W'·x + b'

Two affine maps in sequence collapse into one (W2(W1x) = (W2W1)x).

One hundred such layers? Still one matrix. Five hundred layers of pure matrix multiplies = one expensive linear model that took 500x the time to compute.

So without nonlinearities, depth is useless.

The activation functions between layers are what make composition strictly more expressive.

Which also previews the punchline: the interesting design questions become

• which nonlinearity to insert (topic 77),
• where the gradient dies during training (topics 85-86),
• and how many folds you can afford.

Sanity check with tiny numbers: y = 3·(2x) + 1 = 6x + 1. Two "layers" (×2, then ×3), one line. You cannot fold 6x + 1 into anything but another line. The max(0, ·) twist is what lets a crease exist at all.

Carry one analogy through the rest of this topic: a restaurant kitchen brigade.

• The raw ingredients arrive unorganized: whole fish, dirt-covered carrots. That is your raw input vector — pixels, tokens, features.
• The prep cooks are the hidden layers. They do not plate anything. They perform transformations: fillet, dice, reduce. Each cook takes what the previous station produced and restates it in a form the next station can use.
• The head chef is the output layer. She only makes the final call — "this plate is ready / not ready" — and she can make it easily precisely because the prep cooks already did the hard restructuring.

Now watch each technical claim map onto the kitchen:

• "Hidden layers are learned changes of coordinates" → the prep station invents its own cuts; nobody told the cook that dice-5mm beats dice-8mm, the recipes (training data) forced that skill.
• "Linear layers without activation collapse" → ten workers who each only move boxes to the left counter accomplish what one worker could. No transformation, no depth benefit.
• "Depth buys exponential regions" → each extra station can combine previous prep in ways no single cook could reach: julienne-then-blanche-then-fold.
• "MLPs memorize small datasets" → a kitchen with one recipe card and 50 cooks starts inventing reasons to be busy — overfitting is organizational, not personal.

And the punchline connection: in every transformer, the MLP block after attention is exactly this prep station — attention decides which ingredients to gather, the MLP chops them into new representations the next layer consumes.

An MLP is a rectangle factory, and every rectangle costs exactly:

Count the parameters of a 784→256→128→10 MLP on MNIST (784 = 28×28 flattened pixels, 10 = digit classes):

code
Copy
layer 1: 784×256 + 256  = 200,960 + 256   = 201,216
layer 2: 256×128 + 128  =  32,768 + 128   =  32,896
layer 3: 128×10  + 10   =   1,280 + 10    =   1,290
total ≈ 235,402  → "about 237k parameters" as usually quoted

One number to internalize: a dense layer's cost is the product of fan-in and fan-out. That is why fully-connected layers are avoided on raw images (see topic 84 for how convolutions exploit structure instead).

Practical shape heuristics in the transformer era:

• Bottlenecks (narrow middle layers) force compression and act as regularizers — the kitchen can't fake a 5mm dice if the basket only fits coarse chops.
• Residual MLP blocks (x + MLP(x)) — the feedforward blocks inside every transformer — let depth help via identity shortcuts, easing optimization: if a station adds nothing useful, the ingredients can simply walk past it unchanged.
• Tabular data: gradient-boosted trees still beat deep MLPs on many mid-size tabular benchmarks; MLPs become competitive at 1M+ rows or when features include high-cardinality categoricals needing embeddings (this is the recipe behind two-tower recommenders).
• Width vs depth: for a fixed parameter budget, deeper-narrower usually folds more regions (Montufar 2014, section 4), but deeper also means more places for gradients to attenuate — the optimizer tax on expressivity.

The PyTorch version of the whole architecture is eight lines:

import torch, torch.nn as nn

model = nn.Sequential(
    nn.Flatten(),
    nn.Linear(28 * 28, 256), nn.ReLU(), nn.Dropout(0.2),
    nn.Linear(256, 128),      nn.ReLU(),
    nn.Linear(128, 10),       # raw logits
)

x = torch.randn(64, 1, 28, 28)   # one batch
logits = model(x)                # (64, 10)
print(sum(p.numel() for p in model.parameters()))  # ≈237k params

Hidden units are not features you designed — they are learned representations. Nobody codes "detect a vertical stroke"; the loss function forces some unit into that job.

Classic evidence:

• On MNIST, first-layer ReLU units respond to oriented edges; deeper units respond to strokes and digit parts. Layer 1 = "lines", layer 3 = "curves", layer 5 = "loops like in a 6 or an 8" — the classic computer-vision demo where you can literally draw the filters.
• On language, MLP units inside transformers fire for syntactic or semantic patterns — parts of speech, phrase endings, factual associations.

This gives the modern mental model, one line per layer type:

code
Copy
raw data  →  edges/fields  →  parts/patterns  →  concepts  →  linear readout
 input        layer 1           layer 2           layer k        output

Each hidden layer is progressively more abstract, more invariant, more task-shaped than the last.

Why every AI engineer should care, concretely:

1. On-ramp to LLM internals. The feedforward blocks of GPT-class models are MLPs (with GELU instead of ReLU) sandwiched between attention layers — each transformer block's MLP expands the hidden dimension roughly 4x. If you understand this page, you understand half of a transformer layer already; topic 90 finishes the job.
2. Why "representation learning" beat feature engineering. Pre-deep-learning vision pipelines had humans hand-designing SIFT/HOG features. An MLP learns its own creases — better and cheaper at scale.
3. Why structure matters as much as size. Dense layers treat every input pair as equally related, burning n_in × n_out parameters on every pair. CNNs (topic 84) and transformers (topic 89+) win on images/text by assuming locality/sequence; MLPs win once data is already a vector: tabular rows, embeddings, classifier heads on frozen encoders.
4. Where training still surprises you. The architecture can represent the function (universal approximation) yet gradient descent still fails to find it — bad init, saturation, thin data. Expressivity and trainability are two different books; this topic wrote the first chapter, topics 85-86 write the second.