Semi-supervised Learning

You are building an essay grader.

You have 5 essays with expert scores, and 500 essays with no score at all.

Hiring experts to grade all 500 is slow and expensive.

So the question becomes

Semi-supervised learning (SSL) is exactly that attempt. It trains on a small labeled core plus a large unlabeled pool, to cut annotation cost.

In symbols the practical situation is labels for only 1-10% of your data:

• L = {(xᵢ, yᵢ)} — small, human-labeled.
• U = {xⱼ} — huge, unlabeled.
• Goal: total error minimized over both.

Most SSL methods just add an unlabeled term to the normal supervised loss:

J = (1/|L|) Σ ℓ(f(xᵢ), yᵢ) + λ · (1/|U|) Σ ℓ(f(xⱼ), ŷⱼ)

Read it piece by piece.

• The first half is ordinary supervised loss on the labeled set — the anchor of correctness.
• The second half is loss on unlabeled data, where ŷⱼ is a pseudo-label: a target the model made up for itself. It can be generated (a pseudo-label), implied (manifold smoothness), or replaced by a consistency constraint.
• λ starts small and ramps up. Training hard on your own confident-but-wrong guesses early is exactly how SSL collapses.

Everything rests on three assumptions, and knowing them tells you when SSL will fail:

1. Cluster / continuity assumption: points in the same dense cluster share a label. If your classes are interleaved (overlapping distributions in feature space), propagating labels across a cluster is actively harmful.
2. Manifold assumption: the decision boundary should thread through low-density regions; moving along the data manifold should not change the label.
3. Smoothness / regularization assumption: small input changes (and augmentations) should not change the prediction — the basis of consistency training.

Self-training is the simplest SSL. Four steps, with numbers.

1. Train on the 5 labeled essays. The model now has an opinion.
2. Predict labels for all 500 unlabeled essays. The model says "essay #77 scores 9/10 with 95% confidence."
3. Keep only the confident guesses as pseudo-labels — say the 200 essays it is sure about. Throw away the other 300 for now.
4. Retrain on 5 + 200 = 205 essays. Repeat, and each round 300 more essays get confidently labeled.

In one line

Now the fork in the road.

• If the guesses are mostly right, you just saved hundreds of expert-hours.
• If a confident guess is wrong, you have fed the model its own mistake — and next round it trusts that mistake as if it were truth.

That single failure mode has a name: confirmation bias.

SSL only helps when classes form clean clusters with empty gaps between them. Look at the two cases.

code
Copy
CLUSTER ASSUMPTION HOLDS              CLASSES INTERLEAVED (SSL HURTS)
  ● ● ●        ▲ ▲ ▲                    ● ▲ ● ▲ ●
  ● ● ●        ▲ ▲ ▲                    ▲ ● ▲ ● ▲
  dense A      dense B                   ● ▲ ● ▲ ●   one mixed blob
       ┊                                    (no gap for a boundary
        boundary threads here               to thread cleanly)

• Left: the decision boundary slides through the empty gap, so labeling a whole cluster is safe.
• Right: the classes overlap inside the same dense region. Propagating one label drags the wrong class along with it — this is the cluster assumption violated, and SSL actively injects error here.

So before you reach for SSL, ask

Carry this picture through the whole topic: a study group that owns the correct answers to 5 problems, and blank copies of 500 more.

One strong student (the current model) fills in guesses for the blanks. Now the trap:

A good study group checks every guess against the 5 known answers, keeps only the ones consistent with them, and never lets a confident mistake become the new "fact."

A bad study group copies its own wrong guess onto the shared sheet, then studies from that sheet forever — the mistake is now "truth," and every member reinforces it.

That is pseudo-labeling done right versus confirmation bias. The 5 verified answers are your labeled set: they are what stops the whole group from drifting.

Self-training / pseudo-labeling (Yarowsky 1995; softmax prediction in Lee et al. 2013). Train on L, predict on U, assign hard labels to high-confidence predictions, retrain on L ∪ pseudo-labeled U, iterate. Simple, model-agnostic, and already used inside most "auto-relabel" pipelines. Its known failure mode is confirmation bias: the model's errors become training targets it never sees challenged, and confident mistakes amplify. Mitigations: class-balanced pseudo-label selection, threshold scheduling, and discarding pseudo-labels below a calibrated confidence.

Co-training (Blum & Mitchell 1998). Requires two conditionally independent feature views (e.g., page text vs anchor/link text for web pages). Each model labels data for the other, and agreement across views filters bad pseudo-labels. Works when views genuinely exist; silently fails when they overlap.

Tri-training removes the independence requirement by using three classifiers and only exchanging labels when two agree against the third.

Label propagation and label spreading (graph-based). Build a similarity graph over L ∪ U, then diffuse known labels along edges weighted by similarity. Label spreading adds clamping and a Laplacian regularizer, and tolerates noisy initial labels; both scale poorly to tens of millions of points unless the graph is sparse and approximate (k-NN). Practical use: propagating a handful of confirmed fraud incidents across a dense embedding neighborhood.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

sc = StandardScaler().fit(X_l)
Xl, Xu = sc.transform(X_l), sc.transform(X_u)

model = LogisticRegression(max_iter=1000).fit(Xl, y_l)
for epoch in range(10):
    proba = model.predict_proba(Xu)
    conf, pred = proba.max(axis=1), proba.argmax(axis=1)
    tau = np.quantile(conf, 0.6) + 0.02 * epoch      # tighten over time
    keep = conf >= tau
    X_aug = np.vstack([Xl, Xu[keep]])
    y_aug = np.concatenate([y_l, pred[keep]])
    model = LogisticRegression(max_iter=1000).fit(X_aug, y_aug)
    print(epoch, keep.mean(), model.score(X_val, y_val))

# Stop when validation accuracy stops rising — pseudo-labels are saturating

The modern era replaced "trust the argmax" with "demand agreement." Temporal Ensembling and Mean Teacher maintain a smoothed copy of the model (or its predictions) and penalize disagreement between student and teacher on unlabeled inputs — an unlabeled consistency loss with no generated label at all.

MixMatch combines augmentation, sharpened mixing, and entropy minimization. FixMatch (Sohn et al., Google Brain, 2020) simplifies this into one crisp rule: apply a weak augmentation, take the pseudo-label only if confidence exceeds τ = 0.9, and force the prediction on a strong augmentation to match it.

ℒ = ℒ_sup + λ_u · 𝟙[maxₚ p ≥ τ] · CE(f(strong(x)), argmax f(weak(x)))

Reported results: with only a few hundred labeled images on CIFAR-10 the method reached high-80s/low-90s accuracy that previously required orders of magnitude more labels, and on ImageNet with roughly 10% of the labels it surpassed prior semi-supervised results. The headline is label efficiency: an order-of-magnitude fewer human labels for competitive accuracy. UDA (2020) showed the same unsupervised-consistency objective transfers to NLP (back-translation) and sequence tasks.

Practical constraints that papers understate:

• Threshold calibration. τ = 0.9 is meaningless for an overconfident network; temperature-scale or calibrate the base model first, or false pseudo-labels flood in.
• Augmentation quality. Consistency assumes your strong augmentation preserves the label. It does not for fine-grained tasks (a crop can turn "bird" into "background").
• Class imbalance. Argmax acceptance favors head classes; without per-class thresholds, tail classes never get pseudo-labeled.
• Distribution shift. Pseudo-labels on out-of-distribution data teach the model to confidently label garbage.

Semi-supervised learning is a cost-control strategy, so decide it with a label-efficiency curve, not enthusiasm. Plot validation metric against labeled-set size (25%, 50%, 100% of budget) with and without the unlabeled term. If SSL lifts the curve at low budgets, you have bought annotation savings; if the curves converge, the unlabeled pool is adding noise and you should spend the compute on more labels.

Rules of thumb from practice:

1. Representativeness beats volume. A labeled set sampled uniformly at random from the deployment distribution outperforms a 10x larger convenience sample. Google's "ReLabel" and human-in-the-loop systems use model uncertainty to choose what to annotate next (active learning), which usually stacks with SSL.
2. Start with self-training on the strongest model. Most teams get 80% of the benefit from one clean round of high-threshold pseudo-labeling plus human review of accepted examples.
3. Audit pseudo-labels. Sample 200 accepted pseudo-labels and hand-check them. If accuracy on the accepted subset is below your target model precision, the threshold is wrong.
4. Keep the loss split explicit. Log labeled loss and unlabeled loss separately; when unlabeled loss falls while labeled loss rises, consistency is overriding truth.