This series has used a specific vocabulary — some of it borrowed from statistics and machine learning, some of it built case by case across the chapters of Algorithmic Bias. None of these terms are decoration. Each one earned its place by naming a failure mode that recurs across facial recognition, hiring, credit, welfare, and criminal justice. This glossary collects them in one place, defined the way the underlying cases actually earned them, not as abstractions.
Terms From How the Machine Learns
- Algorithm — A word used, misleadingly, as if it named an entire system. It does not. A deployed system is really four separable things: the training data it was shown, the training method used to extract patterns from that data, the trained model in which the pattern is fixed, and the prediction or classification the model returns for a new case. When an automated decision causes harm, blaming "the algorithm" is often too crude — the failure may sit in any one of those four layers, or in the human decision to treat the output as a verdict.
- Overfitting — When a system performs well on the examples it was trained on but fails on new cases that differ from its training. It has learned the exam, not the world. A model overfit to its training population will treat anyone who doesn't resemble that population as an anomaly, even when the anomaly is simply someone the training data never included.
- The Bias Stack — The chain of translation by which the world becomes a model: social history becomes institutional habit; institutional habit becomes recorded data; recorded data becomes labels and proxies; labels and proxies become training signals; training signals become model behavior; model behavior becomes an institutional decision — which produces new data that flows back to the top of the stack and gets learned again. Unfairness is rarely invented at the level of the model. It is usually admitted layer by layer before the model ever sees a training example.
- Proxy — A feature that stands in for something an institution never measured directly, and often never could measure directly or lawfully. A model trained without access to a protected category can still reconstruct its effect through variables correlated with it — a ZIP code standing in for race, a resume gap standing in for a caregiving history. Removing the named category from a dataset does not remove its proxies.
- Systemic bias — Defined by NIST Special Publication 1270 as bias that "arises from the procedures and practices of particular institutions that operate in ways that result in certain social groups being advantaged or favored and others being disadvantaged or devalued." It requires no prejudiced individual anywhere in the chain — it lives in the institutions that made the records, long before any model was imagined.
Terms From When the System Decides
- The accuracy paradox — The finding that a model can be exactly as accurate as its vendor claims and still be the wrong tool for the decision placed on top of it, because accuracy answers only whether the model predicted its target — not whether that target should govern a person's liberty, credit, care, or employment.
- Disparate impact / statistical inheritance — Two names, from two disciplines, for the same structure. A lawyer calls it disparate impact: a neutral rule producing unequal outcomes because the conditions beneath it are unequal. A technologist calls it statistical inheritance: a model absorbing the distributions, absences, and correlations present in its training sample. Both describe a model that is accurate about an unjust past and, by being accurate, projects that past forward.
- The mitigation reflex — The institutional habit of assuming deployment is already settled and treating the only remaining work as making a system less dangerous: tune the model, broaden the dataset, add review. It is the reflex that treats every problem as a mitigation problem, even the ones that are actually permission problems.
- The refusal threshold — The line past which the responsible decision is not to build a system at all, regardless of how well it might later be tuned. A system crosses it when human legitimacy is non-delegable, when the harm is irreversible at human scale, when the system depends on opacity to function, when its risks fall on the classified while its benefits accrue to the institution, or when its premise feeds historical harm forward.
Terms From What Happens After
- Institutional narration — What happens when a system produces harm, the institution explains it afterward, and the explanation gets treated as accountability even though nothing about the outcome can still be changed. COMPAS is the clean example: a vendor's after-the-fact defense of a fairness metric did not make the detentions that had already occurred any less consequential.
- Contestability — A property of system design, not an apology issued after the fact. A system is contestable only when the affected person can do more than understand the decision — they must be able to move it. This is distinct from transparency and explanation, both of which end at understanding.
- The oversight illusion — The mistaken belief that a person's presence in a decision workflow constitutes meaningful oversight. A reviewer who can see an output but has no real authority to reverse it is not a safeguard. They are a witness. Attention is mistaken for control; hope is not a control.
- Recourse — What a person reaches for after harm has arrived: the appeal, the remedy, the route back. Distinct from contestability, which is the architecture that has to exist before the first decision is made so that reaching for anything is possible at all.
- COMPAS — Correctional Offender Management Profiling for Alternative Sanctions. A proprietary risk-assessment tool used in U.S. criminal courts to score a defendant's likelihood of reoffending. ProPublica's 2016 analysis of more than seven thousand Broward County, Florida defendants found Black defendants were roughly twice as likely as white defendants to be wrongly flagged as high risk.
- The architects of trust — The people, inside and outside a deploying institution, whose authority — exercised or not — determines whether an automated system is accountable: the buyers who write responsibility into a contract, the builders who design the override path, the reviewers whose disagreement with the machine must bind, and the regulators, judges, and legislators who hold the perimeter from outside.
Why a Vocabulary Matters
A shared vocabulary is not pedantry. An institution that cannot distinguish a proxy from a protected category, or contestability from mere transparency, will not know which fix actually closes the gap it has found. "We added an explanation screen" answers a contestability problem the way a smoke detector answers a structural fire risk — it tells you something is happening, and does nothing to stop it. If your organization is building or reviewing an AI governance program, this glossary is the shorthand that lets legal, engineering, and compliance teams argue about the same problem instead of past each other — which is exactly the alignment PIA Studio is being built to support.