Supervised Learning

Imagine teaching a friend to tell real coins from fake ones.

You hand them 500 coins. For each coin you say out loud: "real" or "fake."

That is the whole setup. You have examples, and every example comes with the correct answer attached.

Now the only question that matters

Supervised learning is exactly this. A model learns a mapping from inputs to outputs using a dataset of labeled examples.

Formally, it trains a function

f: X → Y

from input/output pairs (x₁, y₁), …, (xₙ, yₙ).

Each label y is the correct answer. It came from a human, a database, or a real-world outcome that arrived later.

The training algorithm searches the space of possible functions (the hypothesis space) for the weights w that make f(x; w) agree with y on the examples it can see.

Under one big hope

The idea in one line

Let's unpack the words piece by piece.

• x = one input (a coin, an email, a house).
• y = the correct answer for it, called the label.
• f(x; w) = the model. It turns an input into a guess, controlled by knobs w (the weights or parameters).
• Loss = a single number for "how wrong is the current guess."
• Training = keep nudging w so the loss gets small on the labeled examples you can see.

Here is the catch. A model can cheat.

Memorize every training label, and you score perfectly and generalize not at all.

So supervised learning is defined less by "has labels" than by explicitly optimizing expected error on unseen data. Two numbers, two names:

• Training (empirical) risk: average loss over the labeled examples you own.
• Generalization (true) risk: expected loss over the full data distribution you will face in production.

The gap between those two numbers is bounded by model complexity and dataset size.

That is why every serious supervised project spends more energy on the evaluation protocol than on picking a model.

Let's make it concrete with tiny numbers.

Regression predicts a number. Say house price from size.

code
Copy
size (sqft):   500    1000    1500    2000
price ($k):    150     300     450     600

A simple fit could be price ≈ 0.3 × size. For a 1000 sqft home it guesses 300 — dead on. You measure the error on every row, nudge the slope, and repeat. That is the loop.

Classification picks a category. Say spam or not-spam for each email.

Now meet the classic trap. Imagine a fraud model where only 0.2% of transactions are actually fraudulent.

It flags nothing, catches zero fraud — and still scores 99.8% accuracy. Accuracy is now meaningless. Precision-recall curves become the contract.

Three engineering consequences of the split matter more than the theory:

1. Class imbalance. Rare positives make accuracy lie. Reach for precision-recall instead.
2. The threshold is a business decision. Classification hands you a probability; the cutoff (0.5 by default, rarely correct) is set by the cost ratio of false positives to false negatives.
3. Delayed labels. In credit, churn, and ad conversion, the label arrives days after the feature snapshot. Feature/label time leakage is the single most common production bug.

Picture the data as three separate rooms.

code
Copy
   TRAIN            VALIDATE             TEST
   (fit the knobs)  (pick the model)     (final truth)
   ┌─────────┐      ┌─────────┐         ┌─────────┐
   │  70%    │  →   │   15%   │    →    │   15%   │  opened ONCE
   └─────────┘      └─────────┘         └─────────┘
    learn here        tune here          honest score here

The standard discipline:

• Train set — used to fit weights.
• Validation (dev) set — used repeatedly to pick hyperparameters, features, and architectures. Reuse it heavily and it silently becomes optimistic.
• Test set — touched once, for the final claim. After you look at it, it is effectively just another validation set.

When data is scarce, k-fold cross-validation replaces the single split: train k models, each holding out a different 1/k slice, and average the scores. That shrinks the variance of your estimate and exposes instability.

• Use stratified folds for imbalanced classification.
• Use time-series splits (train on the past, test on the future) whenever features have temporal order.

Group isolation is the other non-negotiable rule. If the same user, patient, or house appears in both training and validation, the model can recognize the entity instead of learning the pattern — and your offline score will not survive contact with reality.

The pipeline below keeps that promise: imputing and scaling happen INSIDE the pipeline, so fit() only ever sees the training folds.

from sklearn.model_selection import StratifiedGroupKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

# Impute/scale INSIDE the pipeline so fit() only ever sees training folds
pipe = make_pipeline(
    StandardScaler(),
    LogisticRegression(C=0.5, max_iter=2000),
)

cv = StratifiedGroupKFold(n_splits=5)
scores = cross_val_score(
    pipe, X, y, cv=cv, groups=customer_id, scoring="roc_auc"
)
print(f"{scores.mean():.3f} +/- {scores.std():.3f}")  # spread = fragility signal

Carry one picture through the whole topic: a student preparing for a final exam.

• The textbook problems with answers = the training set. You learn against them.
• The mock exam = the validation set. You use it over and over to choose what to study and which method to keep.
• The real final, sealed until exam day = the test set. You open it exactly once. That number is the honest one.

Now compare two students.

On the textbook, both ace it. On the sealed final, Student A falls apart.

That is overfitting versus generalization — the entire reason supervised learning obsesses about unseen data.

And just like a student, a model can cheat without meaning to. Glancing at the final questions is test leakage. Studying practice problems whose answers were derived from the very thing you are tested on is time or group leakage. The whole evaluation contract exists to stop those two cheats.

Where it wins. Supervised learning dominates when three conditions hold: the target is well defined, labels are obtainable at realistic cost, and the relationship between features and label is stable over time. Pricing, demand forecasting, spam filtering, document routing, medical triage from structured labs, and ad conversion prediction are all strong fits.

The models are often shallow. Logistic regression still runs a large fraction of the world's production predictions because it is cheap, interpretable, and auditable.

Where it does not.

• Labels are expensive or contradictory (radiology annotation, content moderation). See semi-supervised and self-supervised learning in topics 42-43.
• The label distribution shifts. A pandemic rewrites grocery demand; a new feature changes churn behavior. The model optimizes a world that no longer exists.
• The task is actually unstructured discovery. "Tell me segments in my user base" has no label, so it belongs to unsupervised learning (topic 41).
• Feedback loops distort the data. Your model's own predictions determine which items get clicked, and those clicks then become training labels. Off-policy correction or randomized exploration traffic is required.