Regression vs Classification

You run a delivery company and you want a model to help.

A manager asks two different questions.

Same data. Two totally different jobs.

Regression answers the first: it predicts a quantity on a continuous scale (price, temperature, seconds to failure, units demanded).

Classification answers the second: it predicts a discrete category (spam/not spam, late/on-time, disease present/absent, transaction type).

This looks like a small distinction. It is not. The type of your target y is not a cosmetic detail — it decides almost everything downstream.

The rule in one line

Because y looks different, five things change together. The target type determines:

1. Output layer / hypothesis form: an unbounded linear combination versus a squashed probability or class score.
2. Loss function: squared error is the maximum-likelihood objective under Gaussian noise; cross-entropy is the maximum-likelihood objective under a Bernoulli/Categorical model. Optimizing the wrong loss means optimizing the wrong thing even when the model class is capable.
3. Metric — and therefore what "good" means: RMSE versus ROC-AUC behave completely differently under imbalance and outliers.
4. Whether a decision exists: classification adds a threshold step that regression does not have, and that threshold is where business cost enters the pipeline.
5. Data requirements: learning a decision boundary between two heavily overlapping classes generally needs less data than estimating a precise conditional mean across the same region, but calibrated probabilities demand more.

A useful mental model to keep

Probability estimation is itself a bounded regression problem on [0,1], which is why the two worlds borrow each other's machinery constantly.

Take a single measurement — delivery delay in minutes — and frame it three ways on the same rows.

code
Copy
order:        A     B     C     D
delay (min):  2     9     20    45

• Regression: predict the delay itself (2, 9, 20, 45...). Being "wrong" means being off by some number of minutes.
• Classification: define late = delay > 15. Now predict a category. C and D are late; A and B are on time. Being "wrong" means picking the wrong pile.
• Threshold with cost: suppose a late delivery costs 4x a false alarm. Instead of the default 0.5 cutoff, you pick the one that minimizes expected business cost.

The script below trains all three framings and prints the metric each one needs:

• Regression → RMSE.
• Classification → ROC-AUC and log loss.
• Asymmetric cost → the best threshold, which is not 0.5.

Notice: the regression never touches a threshold. The classifier does — and that threshold is where the money enters.

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import mean_squared_error, roc_auc_score, log_loss
import numpy as np

# 1) Regression: predict the continuous delay in minutes
reg = LinearRegression().fit(X_train, delay_train)
print("RMSE:", np.sqrt(mean_squared_error(delay_val, reg.predict(X_val))))

# 2) Binary classification: is the delay > 15 minutes?
y_late = (delay_train > 15).astype(int)
clf = LogisticRegression(max_iter=1000).fit(X_train, y_late)
p = clf.predict_proba(X_val)[:, 1]
y_val_late = (delay_val > 15).astype(int)
print("ROC-AUC:", roc_auc_score(y_val_late, p), "log loss:", log_loss(y_val_late, p))

# 3) Asymmetric cost: pick the threshold that minimizes expected business cost
late_cost, false_alarm_cost = 4.0, 1.0     # missing a late delivery costs 4x
thresholds = np.unique(p)

def expected_cost(t: float) -> float:
    pred = (p >= t).astype(int)
    fn = np.sum((pred == 0) & (y_val_late == 1))   # late but not flagged
    fp = np.sum((pred == 1) & (y_val_late == 0))   # on time but flagged
    return late_cost * fn + false_alarm_cost * fp

best_t = thresholds[int(np.argmin([expected_cost(t) for t in thresholds]))]
print("cost-optimal threshold:", best_t, "vs default 0.5")

The two jobs grade "being right" in two different pictures.

code
Copy
REGRESSION — a number line, measure the distance
   0──────10──────20──────30  minutes
        ▲ pred=12     ● actual=20        error = |20 - 12| = 8 min

CLASSIFICATION — a gate, did you land in the right box?
   [ ON TIME │ LATE ]
            p = 0.7  ── threshold 0.5 ──►  call it LATE
   actual was LATE  →  right box ✓   (just right / wrong, no "how far")

Regression asks "how far off was the number?" — a magnitude, so the loss squares or abs the gap.

Classification asks "which box, and were you right?" — a category, so the loss rewards the probability and a threshold makes the final call.

That single difference is exactly why their losses and metrics cannot be swapped.

Think of a weather app on your phone.

The temperature forecast is judged by how many degrees it missed — a magnitude (RMSE). The rain forecast is judged by whether the yes/no matched.

Now the part that trips people up: to act on the rain forecast you need a threshold.

That cutoff is a business decision, not a math fact — exactly like the delivery-cost example. Set it too low and you cry wolf; too high and people get soaked.

And a probability is the bridge between the two worlds. "60% chance of rain" is a number (regression-like) that you turn into a call (classification-like). This is why the two problems share machinery.

Regression losses differ in how they treat error magnitude and asymmetry:

• MSE / squared error penalizes large errors quadratically, is smooth (easy gradient descent), and corresponds to maximum likelihood for Gaussian residuals. Sensitive to outliers.
• MAE / absolute error is robust to outliers, corresponds to a Laplace error assumption, and has a discontinuous gradient at zero.
• Huber interpolates: quadratic near zero, linear in the tails — the default choice when a handful of extreme values are real but must not dominate.
• Quantile (pinball) loss estimates conditional percentiles, which is how you produce prediction intervals and how you encode asymmetric business costs (under-forecasting stockouts is worse than over-forecasting).

Classification losses: binary cross-entropy / log loss rewards probabilities and is properly scoring; hinge loss (SVM) only cares about margin and ignores calibration; 0-1 accuracy is non-differentiable so it can only be a metric, never a training objective.

The metric mismatch that ruins models: training on log loss but shipping on hard accuracy at a fixed 0.5 threshold discards both the ranking information and the cost asymmetry. Match objective to decision:

• Imbalance-heavy (fraud, defects): average precision / PR-AUC and precision-recall trade-offs.
• Balanced, ranking-focused (search, credit ranking): ROC-AUC, with the caveat that ROC is optimistic under extreme imbalance.
• Probability must be trusted (risk-based pricing, clinical triage): Brier score and calibration curves, not just discrimination.

Teams routinely re-express a problem, and each conversion has consequences.

Regression → classification (binning/discretization). Cut a continuous target at a business-relevant threshold: "revenue > $100", "readmission within 30 days". You gain a crisp actionable output and often a more learnable signal, because predicting the direction is easier than the exact number. You lose information at the boundary, and points just above and just below the cut are treated as maximally different — a patient recovering in 31 days is labeled with the discharged-in-2-days. Choose the cut from cost data, not convenience.

Classification → regression (probability as a continuous target). A calibrated P(y=1|x) is a regression onto [0,1] and unlocks expected-value decisions: rank customers by p·value, price risk by expected loss. This is why large-scale ad systems optimize predicted CTR (a probability) but bid on expected value.

Count data is neither. Item demand, click counts, and insurance claims are non-negative integers with variance growing with the mean; ordinary least squares on raw counts produces negative predictions and wrong standard errors. Use Poisson or negative binomial regression (log link) — or regress on log(1+y) and remember to correct when exponentiating predictions (Jensen's inequality: mean of exp(log y) ≠ exp(mean log y)).

Ordinal categories are not nominal. "Low/Medium/High" risk has order; one-hot encoding it and using multiclass cross-entropy ignores that structure, so a swap between Low and High costs the same as Low vs Critical. Better: regression on integer codes with monotonic constraints, or cumulative-logit modeling that estimates P(y ≤ k) for each cut.

Multiclass is k binary problems in disguise. Softmax cross-entropy, one-vs-rest, and one-vs-one are three legitimate decompositions; for very large label sets (thousands of tags), one-vs-rest with shared features or embedding-based retrieval outperforms softmax over the full vocabulary.

The regression/classification split survives all the way into the deployment architecture.

• Output contract. A regression service returns a number plus an uncertainty band; a classification service returns a probability plus a threshold policy. Consumers of the latter must be told what the threshold is and who owns changing it — otherwise a silently static 0.5 governs a million decisions.
• Monitoring. Regression drifts visibly (mean prediction, residual distribution, RMSE against delayed actuals). Classification needs both score-distribution monitoring and label-arrival monitoring; if 30-day outcomes lag, the error estimate you report this week was computed on last month's cohort.
• Intervention design. Regression supports continuous levers (adjust price by Δ). Classification supports binary gates (approve/decline), which makes thresholds the main knob and demands periodic cost re-estimation.
• Fairness and regulation. Credit and medical decisions often require explanation of a score; logistic regression and gradient-boosted models with monotonic constraints are preferred over opaque ones because regulators ask for reasons, and classification decisions get appealed while regression estimates usually do not.

Choose the framing by asking the decision question first: "What will someone do differently because of this output?" If the action is a yes/no gate, classify. If it is a magnitude adjustment, regress. If it is "sort the top 5% to an agent queue," both work and you should pick whichever is more learnable and calibratable.