Entropy (Information Theory)

"The sun rose today." Almost no news.

"It snowed in the Sahara." Huge news.

Notice the pattern:

But "carries information" is fuzzy. Engineers need a number. And a good one must behave:

• If something is certain (p = 1), it should teach you nothing → information = 0.
• If something is nearly impossible (p → 0), it should be mind-blowing → information → ∞.
• Two independent surprises should add: learning "it snowed in Sahara AND I won the lottery" costs the sum of the two shocks, even though their probabilities multiply.

So the question becomes

Shannon had one: logs. (Topic 24: log turns products into sums.)

Surprisal (self-information) is simply

Unpack it:

• Monotone: rarer events are more surprising. As p → 0, I → ∞. As p → 1, I → 0. Check and check.
• Additive for independent events: I(x and y) = I(x) + I(y), because probabilities multiply and log(ab) = log a + log b.
• And it is the only functional form satisfying both properties, up to a constant. The requirements force the log.

The base of the log sets the unit:

• log₂ → bits (how many binary digit flips / yes-no questions the event answers).
• ln → nats (the natural unit in ML, because derivatives of exp/ln are clean).
• log₁₀ → hartleys.
• In practice: 1 nat = 1/ln 2 ≈ 1.44 bits, and most ML libraries report nats.

Price some real events.

• "Sun rises" (p = 1): I = −log₂ 1 = 0 bits. No news.
• Fair coin lands heads (p = 0.5): I = −log₂ 0.5 = 1 bit. Exactly one yes/no question.
• Fair die shows a 4 (p = 1/6): I = −log₂(1/6) = log₂ 6 ≈ 2.585 bits.
• Snow in Sahara (say p = 0.001): I = −log₂ 0.001 ≈ 9.97 bits — worth about ten coin flips of news.

See the arithmetic of surprise:

code
Copy
p = 1      →  I = 0 bits      ("of course")
p = 1/2    →  I = 1 bit       ("huh, could go either way")
p = 1/4    →  I = 2 bits      ("wait, what?")
p = 1/1000 →  I ≈ 10 bits     ("!!!")

Halving the probability always adds exactly 1 bit. That is what "bits" means here: each bit answers one yes/no question, and each step down in likelihood needs one more question to explain.

One event has surprisal. A whole distribution keeps surprising you at different rates. So average the bill:

H(X) = E[I(X)] = − Σₓ p(x) log p(x) (discrete)

Read it three equivalent ways:

1. The average number of bits needed to encode one draw, if you code optimally.
2. The uncertainty remaining about X before you see it (the average surprise of what you will learn).
3. A property of the distribution alone — H is a function of p, not of any observation.

Two anchor cases to burn in:

• Deterministic: p = (1, 0, …): H = 0 — nothing can surprise you.
• Uniform over k values: H = log k — the maximum; every outcome is maximally unpredictable.

A fair coin has H = 1 bit: one yes/no question describes it exactly.

Two structural facts: entropy is concave (mixing distributions raises it) and chain-rule decomposable: H(X, Y) = H(X) + H(Y | X) — the basis for mutual information (section 6).

import numpy as np

def entropy(p, base=2.0):
    p = np.asarray(p, dtype=float)
    nz = p[p > 0]                      # 0 * log 0 := 0 by continuity
    return -(nz * np.log(nz)).sum() / np.log(base)

for q in [0.0, 0.1, 0.5, 0.9, 1.0]:
    print(f"coin P(heads)={q:.1f}  H = {entropy([q, 1 - q]):.4f} bits")
# 0.0000 at the extremes, 1.0000 for the fair coin: max is log2(2) = 1

print("dice (uniform over 6):", round(entropy([1/6] * 6), 4), "bits")  # log2(6) ≈ 2.585

Picture entropy of a coin as a function of its bias. It is a dome — zero at both certainties, peaked at fair:

code
Copy
 H (bits)
   1 ┤            ●  p = 0.5: "I truly cannot call it"
     ╱ ╲
 0.5┤   ●        ●   p = 0.9 or 0.1: mostly knowable
     ╱    ╲    ╱
   0┤●──────●───────  p = 0 or 1: total certainty, zero surprise
     └──────┴───────►  p (chance of heads)
     0      0.5      1

Carry one analogy through the rest of the topic: a telegraph office that bills you by surprise.

• The office knows your news feed's frequencies.
• Routine messages ("sun rose") get a short code — cheap, few bits.
• Rare messages ("Sahara snow") get a long code — exactly as long as their surprisal.
• Your average monthly bill = the entropy of your news feed.

The genius move: bill each event I(x) = −log p(x) and the arithmetic adds up across independent events. The average bill is minimized when the codes match reality's frequencies — which is Shannon's source coding theorem, next.

Shannon's noiseless coding theorem, in plain words:

Optimal codes assign exactly I(x) = −log p(x) bits to symbol x: frequent symbols get short codewords, rare ones get long codewords (the telegraph's pricing is the optimal dictionary).

• Huffman coding achieves within 1 bit of H.
• Arithmetic coding approaches H arbitrarily closely in practice.

That is why entropy is measured in bits and called "information": it is the minimum description length, the true size of the distribution's content.

Model-based compressors (ZIP-family tools, or using an LLM to compress text) win by predicting better — smaller surprisals under a sharper model. Which is exactly why "compression equals intelligence" arguments connect entropy to modeling quality: a better model = cheaper telegraph bill.

Joint and conditional entropies extend the idea:

• H(X, Y): the cost of coding both.
• H(Y | X): what remains of Y's surprise after X is known.
• Their difference, mutual information I(X; Y) = H(Y) − H(Y | X), counts exactly how much knowing X reduces uncertainty about Y — used for feature selection and representation-learning objectives.

For continuous variables, the discrete sum becomes an integral — differential entropy H = −∫ f log f. Three warnings before you trust it:

1. It can be negative. A concentrated density has little uncertainty: a Gaussian with σ = 0.01 has differential entropy ≈ −3.2 nats.
2. It is not invariant to unit changes: rescaling by c adds log c.
3. The uniform-over-[0,1] case is 0 nats — density ≠ probability.

One encouraging fact: the Gaussian maximizes differential entropy among all distributions with fixed variance, at H = ½ log(2πeσ²) — one more reason the Normal is the "assume-nothing-extra" default.

Where entropy actually appears in your ML day-to-day:

• Decision trees (ID3/C4.5/sklearn): pick the split maximizing information gain = H(parent) − Σ (weighted H(children)).
• Exploration in RL: entropy bonuses (the PPO entropy term, Soft Actor-Critic) reward policies whose action distributions stay uncertain, delaying premature convergence.
• Confidence diagnostics: prediction entropy over classes flags uncertain inputs; attention entropy measures spread of attention; temperature scaling changes output entropy without changing the argmax.
• Regularization: maximize output entropy to prevent mode collapse — a deterministic MLE fit can otherwise go all-in on wrong labels.