Expectation & Variance

Topic 15 gave you the scorekeeper: a random variable turns chaos into numbers, and a PMF or PDF tells you how likely each number is.

Beautiful — but overwhelming.

A model's loss per example, a user's latency, the clicks per visit: each has a whole curve of possibilities attached.

Everyone asks these two questions. Statistics answers them with two numbers:

• Expectation → the average, weighted by probability. The center.
• Variance → the typical squared distance from that center. The spread.

Almost everything in this course — training loss, gradient noise, the bias-variance tradeoff — is these two numbers wearing different hats.

Two formulas, one per world:

• Discrete: E[X] = Σ x·p(x) — multiply each value by its probability, add them up.
• Continuous: E[X] = ∫ x·f(x) dx — same idea with the density's area instead of lumps.

"Weighted by probability" just means: values that happen more often pull the average harder.

You can also take expectations of transformed variables:

E[g(X)] = Σ g(x)·p(x) (or the density integral) — you average the transformed values. This has a joke of a name: LOTUS, the "Law of the Unconscious Statistician."

Physicists say it best: expectation is the center of mass of the distribution. Balance the sand pile of Topic 15 on your fingertip — the spot where it doesn't tip is E[X].

Roll a fair die. Values 1–6, each with probability 1/6.

Step 1 — expectation.

E[X] = (1 + 2 + 3 + 4 + 5 + 6)/6 = 21/6 = 3.5

Notice 3.5 is impossible to roll — the callout above made the point for a reason. It's the balance point, not a prediction.

Casino version: a game pays you ₹10 × the number rolled. Average payout = 10 × 3.5 = ₹35 per play. If the table charges ₹40 to play, the house's average profit is ₹5 per round — casinos are just expectation accountants.

Step 2 — variance. Spread is the average squared distance from the center:

Var(X) = E[(X − E[X])²]

The shortcut formula (algebra expands the square):

Var(X) = E[X²] − (E[X])²

• E[X²] = (1 + 4 + 9 + 16 + 25 + 36)/6 = 91/6 ≈ 15.17
• (E[X])² = 3.5² = 12.25
• Var(X) ≈ 15.17 − 12.25 = 2.92 (exactly 35/12)

Step 3 — standard deviation. σ = sqrt(Var) ≈ 1.71 — the square root puts the units back on the die (pips, not squared pips).

Squaring does two jobs: it kills minus signs (a 1 and a 6 are both 2.5 away, in opposite directions — naive averaging would say "zero spread"), and it punishes far-out values extra.

One identity makes expectations useful at scale:

E[aX + bY] = a·E[X] + b·E[Y]

Constants pass through. Sums split. Coefficients distribute.

And here's the magic part:

X and Y can be tangled together — correlated, causal, from the same process — and the split still works. That single fact makes expectations tractable for sums of hundreds of dependent random variables — a big reason probabilistic models and RL value functions (the "expected future reward" of Q-learning) are computable at all. Variance can't do this (watch the cross term in the next section); expectation can.

Classic illustration — the hat-check problem: n people hand hats to a counter, hats are returned randomly. Expected number who get their own hat back?

• Write it as a sum of n indicator variables: Iᵢ = 1 if person i matches, 0 otherwise.
• Each person matches with probability 1/n, so E[Iᵢ] = 1/n.
• Linearity (the indicators are horribly dependent — forget it!): E[total] = n × 1/n = 1.

A random permutation of n items has, in expectation, exactly 1 fixed point — for any n, even a million. No independence, no integrals, one line.

Variance is tamer-looking but does demand care about dependence. Rules worth memorizing:

• Var(aX + b) = a²·Var(X) — shifting (adding b) does nothing to spread; scaling stretches it quadratically (double the units, quadruple the squared spread).
• Var(X + Y) = Var(X) + Var(Y) + 2·Cov(X, Y) — the extra cross term is exactly the part that independence (or at least zero covariance) kills. Two variables that rise together make the sum more variable than you'd guess.
• Covariance Cov(X, Y) = E[(X−E[X])(Y−E[Y])] tracks joint movement: positive = they drift up together, negative = one rises as the other falls. The correlation coefficient ρ = Cov/(σₓσ_y) rescales it to [−1, 1] so units stop mattering.
• Zero covariance does not imply independence in general (only jointly Gaussian families make that step safe). Uncorrelated can still be deeply related — see the quiz.
• The covariance matrix Σ collects all pairwise covariances of a random vector; it is the second-moment object behind PCA, whitening, and Gaussian distributions.

import numpy as np

rng = np.random.default_rng(0)
true_mean = 0.5                       # mean of Uniform(0, 1)

for m in [10, 100, 10_000, 1_000_000]:
    samples = rng.uniform(0, 1, size=(2000, m))
    est = samples.mean(axis=1)                    # 2000 independent estimates
    se = est.std()                                # empirical standard error
    theory = np.sqrt((1/12) / m)                  # sqrt(Var(X)/m)
    print(f"m={m:>8}  bias={est.mean()-true_mean:+.4f}  SE={se:.4f}  theory={theory:.4f}")

Every dartboard shows the mean (a dart sticks in exactly one place per throw), but the board remembers the whole throw pattern:

code
Copy
   AIM = mean        AIM = mean
       │                  │
   ·  ·│·  ·           ·  ·                  ·
       │ ·  ·              ·        ·           ·
   spread = variance  ·   │   ·       · ·        ·
   small (tight)          ··              ·    ·
──────────┼──────────  ──────┼──────────────
       │                      ·
       ● throws cluster       ● throws scattered wide

• Same center, different spread: that difference is variance.
• Expectation = where the darts cluster on average.
• Variance = the size of the cloud around it.
• The code block above shows the third act: average m throws together and the cloud shrinks like 1/sqrt(m) — the standard error. Batch size in ML does exactly this to your gradient.

Carry one analogy: a random variable is an archer, and each draw is one shot.

• Expectation is the archer's aim — the spot on the scoreboard all the shots average out to (which, like 3.5 on a die, no single arrow ever hits).
• Variance is the grouping of the arrows — tight cluster or scattered across the target. Great aim with terrible grouping still loses tournaments.
• Covariance asks: when the wind pushes the left hand, does the right hand drift the same way? Two archers marching together add their wobble; perfectly opposite ones cancel it.
• Bias (coming in section 8) is the coach's word for a systematic aim error: the sight is misaligned, and every shot is off in the same direction. Averaging more shots won't fix bias — only re-sighting will. But averaging does fix bad grouping (÷m variance, ÷√m standard error).

Every idea in the rest of this topic is one sentence about this archer. Keep the target in view when formulas appear.

Law of total expectation (the tower rule): E[X] = E[ E[X | Y] ] — average within each group, then average the group-averages. Mini-batch training uses it implicitly: the expected batch loss equals the population loss.

Law of total variance: Var(X) = E[Var(X | Y)] + Var(E[X | Y]) — total spread = average within-group spread plus between-group spread (in dart terms: how scattered the archer is around her own aim, plus how much her aim itself wanders between days). It powers hierarchical models, analysis-of-variance reasoning, and variance-reduction tricks (conditioning on a control variate lowers Monte-Carlo noise).

Finally: ML is always estimating a truth θ from data. An estimator T is judged by exactly this vocabulary:

• Bias: E[T] − θ (systematic offset — the misaligned sight).
• Variance: Var(T) (how much the estimate jitters across datasets — loose grouping).
• MSE: E[(T − θ)²] = Bias² + Variance — the Pythagorean identity that is the bias-variance tradeoff. You can trade one for the other, never escape both.

Housekeeping fact with teeth: sample variance with the n−1 denominator (Bessel's correction) is unbiased; the MLE variance with denominator n is biased low by a factor (n−1)/n — a preview of Topic 19's caveats.

The archer is everywhere once you look:

• Empirical risk minimization: the training objective is literally an expectation, L(θ) = E_(x,y)~p_data [ loss(f(x; θ), y) ], approximated by a batch mean. Gradient noise has variance σ²/m, which is why batch size acts as a temperature knob on SGD (m darts averaged → cloud shrinks by √m).
• The ML bias-variance decomposition for squared-error prediction: expected error = bias² + variance + irreducible noise σ² — statistical estimation language applied directly to model selection, overfitting, and ensembling (bagging cuts variance — average many high-skill scatter-brained trees; boosting can cut bias — correct the systematic aim error round by round).
• Normalization layers: BatchNorm standardizes activations to zero mean, unit variance using exactly these two statistics.
• Policy gradients in RL: the objective J(π) = E[return] is an expectation over trajectories; baselines are chosen to reduce its variance without adding bias.
• Uncertainty estimation: Bayesian models and ensembles output predictive means and variances so systems can reason about confidence, not just point answers.

In practice, your dashboard should show both: a metric's level (expectation) and its jitter (variance). A model whose average loss is fine but whose per-batch variance has exploded is an archer whose grouping just went wild — something is wrong upstream.