Bias and Fairness in ML Systems
ML systems inherit unfairness through four doors — history, data coverage, faulty proxies, and one-size-fits-all thresholds — and the three formal definitions of "fair" provably contradict each other. This topic explains where bias enters, what demographic parity, equalized odds and calibration actually mean, why language models add their own failure modes, and how slice-level evaluation plus documentation makes fairness measurable instead of rhetorical.
01.The Problem: A Crooked Ruler Measures Crookedly
A bank trains a model on ten years of loan decisions. The model learns beautifully. Everyone celebrates — except that some neighborhoods almost never get loans, and history shows the old decisions were partly prejudice, not math.
The model didn't invent the unfairness. It measured the world with a ruler the world had already bent, and then froze that ruler into software used on millions of people.
This is the core problem of ML fairness, and it hides in four places (the classification now standard, from Barocas & Selbst, Google's technical ML fairness documentation, and Weidinger et al. for language models):
- Historical bias. The world genuinely is unfair, and the label encodes it. Predicting "arrest within 2 years" or "loan default" reproduces past decisions; predicting re-arrest as a proxy for risk imports policing intensity and charging practices straight into the target.
- Representation bias. Subgroups are under-represented or distorted in the data. In NLP this dominates: web corpora over-represent some languages, dialects, names and cultural frames, so models handle AAVE, non-Latin scripts, and Global-South contexts less well — partly a tokenizer problem too (rare scripts fragment into many subwords, costing effective capacity).
- Measurement bias. The variable you operationalized is not the thing you care about. Using credit utilization as a proxy for creditworthiness, published-paper counts for "research quality", or symptom codes for "need for care" — the Obermeyer et al. 2019 finding is the canonical case: a healthcare risk algorithm trained on cost systematically under-scored Black patients who were sicker at equal cost. That is a construct-validity failure no fairness metric can repair after the fact.
- Aggregation bias. One global model or threshold applied to populations whose relationships actually differ. Fitting one global accuracy curve can be locally wrong for every group.
And then there is feedback: a deployed score changes the world it predicts (higher-risk classification → more patrols → more recorded offences → higher predicted risk...). That loop is why post-deployment monitoring is part of fairness engineering, not an afterthought.
Bias Sources → Competing Fairness Criteria → Governance ⚖️
Bias Sources → Competing Fairness Criteria → Governance ⚖️
Bias enters through history, representation, measurement and aggregation. The three families of formal fairness criteria cannot all hold simultaneously when base rates differ, so a fairness decision is a documented value choice, not a metric to maximize.
Unlock Topic #246: Bias and Fairness in ML Systems
You are viewing a preview. The full in-depth technical walkthrough, worked derivations, and code notebooks for this concept, along with self-assessment quizzes, are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and actionable was this distributed systems breakdown?