Loss and Cost Functions

A model does not optimize "accuracy." It optimizes one specific math object: the cost function.

Every design decision hides inside it — what you punish, how hard, and how far you are willing to shrink the weights.

Get the cost wrong and the model learns the wrong lesson — perfectly, with great training scores, and useless in production.

So first, three terms people routinely mix up:

• Loss ℓ(ŷ, y) — the error on ONE example. Squared error, log loss, hinge loss.
• Cost J(w) — what the optimizer actually minimizes: the average loss over the training set, plus a penalty on the parameters:

J(w) = (1/n) Σᵢ ℓ(f(xᵢ; w), yᵢ) + λ·Ω(w)

• Metric — the number reported to humans: accuracy, RMSE, ROC-AUC, MAPE, average precision. Metrics may be non-differentiable, and they are frequently not the loss.

Train on log loss, judge by accuracy at a fixed 0.5 threshold, and you are optimizing one thing while grading the model on another.

Two clarifications worth keeping exact:

• The loss lives on the model's output space, so output layer and loss must pair: linear + squared error, sigmoid + binary cross-entropy, softmax + categorical cross-entropy, unbounded output + Huber for heavy tails.
• Convexity, smoothness, and curvature of J decide whether optimizing it is even easy. Logistic cost is convex but not strongly convex, so the condition number of the Hessian governs convergence speed (topics 49, 51).

Hold one picture in mind: a parking meter. Every mistake ticks a fine, and the fine rule is the loss function. Change the rule, and you change what the driver (the model) does.