The score arrives ahead of the defendant. A booking officer works through a long questionnaire — age, education, employment, residence, family history, prior arrests, drug use. The answers go into a computer. Minutes later, three numbers come back, each between one and ten: the risk of reoffending, the risk of violence, the risk of failing to appear. The defendant has not yet said a word, and the file already contains a verdict about him written in the grammar of probability.
Those numbers travel — from intake to first appearance, from first appearance to bail hearing, and in some matters to sentencing. In the formal language of the system that produces them, they are a decision support tool. No judge is required to defer to them. But in the rhythm of an overscheduled docket, in a hearing that runs less than ten minutes, the score does work the disclaimer says it should not do. The judge has not met the defendant before this morning and will not meet him after it. The score is the most concrete thing in a folder full of uncertainty, and concreteness, under time pressure, is its own kind of authority.
This is COMPAS, the risk model whose errors, as ProPublica showed, fell more heavily on Black defendants. That the tool is skewed is well established — it's the finding this series has already covered. This piece takes up the harder question that survives even after the skew is granted: suppose the number is exactly as accurate as its vendor claims. It still had to travel from a server to a sentence. Someone decided it was allowed to make that journey, and allowed to mean what it came to mean at the end of it. If the model is accurate, who decides what that accuracy is allowed to do?
Accuracy Was Supposed to Be the Cure
When people hear that an algorithm is biased, they usually imagine the cure is accuracy: make the model better, feed it cleaner data, tighten the prediction. That sounds right. It is incomplete. Accuracy answers one question — did the model predict its target? It does not answer the harder question: should that target govern a person's liberty, care, credit, insurance, or employment? A model may predict arrest without measuring crime. It may predict cost without measuring medical need. It may predict default without measuring worth. Each prediction can be useful in the narrow sense and dangerous in the institutional one. The danger begins in the handoff — the short, unexamined step where a number leaves the model, enters an institution, and becomes permission.
A Working Model Can Still Be the Wrong Tool
A recidivism model may predict arrest. But arrest is not the same thing as crime. A health-risk model may predict cost, but cost is not the same thing as medical need. A model may learn that some patients historically generated lower medical costs and rank them as lower risk — but lower cost may mean less access, fewer visits, delayed treatment, or a system that failed to see the patient sooner. The prediction can be accurate against the billing record and still wrong for allocating care. The model did not invent the inequality. It learned the institution's history of attention and turned it into a future rule.
Modern governance frameworks are trying to force that distinction. The EU AI Act treats systems in justice, credit, employment, health care, and essential public services as high-risk because their outputs can change lives. Rights-based laws such as the GDPR, Brazil's LGPD, and Quebec's Law 25 give people review, notice, or a chance to be heard when automated decisions affect them. The point is simple: accuracy may validate a prediction. It does not validate what an institution does with it, because a working model is not the same thing as a legitimate decision.
What the Number Was Allowed to Mean
Most accounts of COMPAS stop too early, treating the case as a story about a biased model. That framing is true, but too small — the deeper failure was not what the algorithm predicted, but what the institution allowed the prediction to mean. The model was not trained to predict crime in some pure moral sense. It was trained to predict a new fingerprintable arrest within a defined period. That is related to crime. It is not the same thing.
An arrest is not an event floating freely in the world. It is the end of a chain of institutional choices — police have to be present, notice the conduct, decide to intervene, make the arrest. Two people can do the same thing and leave different traces in the data. One becomes a record. The other becomes nothing. So when a model predicts that a defendant is likely to be arrested again, it may be predicting something real — but what it is predicting is not simply future criminal conduct. It is predicting the likelihood that the institution will notice, process, and record that person again. Prior arrests are not crimes. They are the institution's record of which crimes it noticed.
Once that distinction disappears, accuracy becomes dangerous. And then the loop tightens: a defendant marked high-risk is watched more closely, supervised more intensely, given less room for error. More attention produces more records. More records confirm the risk. The model appears to improve. The person disappears behind the pattern. That is the accuracy paradox in its sharpest form — the model may be right about what the institution is likely to see again. That does not mean the institution should use the model to decide who deserves liberty, suspicion, or punishment.
Accuracy Becomes a Credential
The harder problem begins when accuracy changes the burden of proof. A validated score can be described as tested, calibrated, audited, and statistically defensible — each claim may be true, but none of them answers the prior question of whether the thing being predicted is the thing the institution should use to decide a person's future. Once a number looks scientific, it stops being treated as a claim to examine and starts being treated as a fact to defer to. Before the score, the institution had to justify what it was doing to a person. After the score, the person has to argue against a number whose workings he cannot see. The institution gains objectivity. The defendant loses standing.
The scandal of COMPAS is not that the model was wrong about a given defendant. The scandal is what the institution decided the model was allowed to mean.
That decision — to let an accurate prediction of one thing stand in for a judgment about another — was not technical, and the algorithm did not make it. It was made before the score reached the courtroom, by the people who approved, bought, deployed, and normalized the system. The technical question — is the model accurate? — does not reach them.
Four Questions Before Deployment
Accuracy cannot decide whether a system should be used. It can only tell an institution whether the model predicts what it was built to predict. Before deployment, four questions have to be answered.
- Accurate at predicting what? Name the literal target. COMPAS predicts arrest, not crime. A care-management tool may predict cost, not medical need. The gap between the target and the real decision is where the danger begins.
- Accurate for whom? Average performance can hide the people most exposed to error. An institution that cannot say who the system works worst for does not yet understand the system it is using.
- Accurate at whose expense? Every error has a direction and a bearer — a false positive may cost one person liberty, care, credit, or work; a false negative may cost the institution very little. The question is who pays for the mistakes.
- If the model works, what future does it reproduce? A model trained on an unjust past can work exactly as designed and still project that past forward. Success, in that case, does not mean correction. It means repetition.
These questions do not reject accuracy. They put accuracy in its place. A performance score can show that a model predicts. It cannot decide whether the institution should give that prediction power over a person. That judgment belongs to the people deploying the system — if they do not make it, the model will make it for them, quietly, faithfully, and in the direction the data already pointed.
The Question That Comes After Measurement
When a model is built, validated, and deployed, its outputs begin to do work inside an institution. Probabilities become permissions. Scores become sentences. Predictions become denials. At that point, the question is no longer whether the model is accurate. The question is who has authority over what the model is allowed to decide. Most institutions, asked that question today, have no clear answer — the data scientist did not deploy the system, the procurement officer did not validate it, and the person at the point of decision inherited a system whose construction they were never trained to see. The accuracy paradox is a problem of measurement. What it uncovers is a problem of authority — and that is precisely the gap a Privacy Impact Assessment for any automated decision system needs to close before deployment, not after.