In the Broward County courts of 2013 and 2014, a number called COMPAS arrived ahead of the defendant and carried the authority of a measurement. COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — produced a risk score meant to predict how likely someone was to reoffend, and courts leaned on it in decisions about detention and sentencing. A defendant who wanted to challenge that number had a real problem: he could not see how it worked. The training data, the weighting of features, the threshold at which "low risk" became "high risk" — all of it belonged to the vendor, Northpointe, later Equivant, as a trade secret.
He did not know what the model weighted. He did not know whether it behaved differently on people who looked like him. His lawyer did not know either. And there was nowhere to take the question — no judge had barred it; there was simply no mechanism to force the model open. The score was unappealable, not because anyone had forbidden the appeal, but because no procedure existed to receive it.
What ProPublica Found
Three years later, outsiders did what the system itself could not. A team of ProPublica journalists — Larson, Mattu, Kirchner, and Angwin — obtained the scores of more than seven thousand Broward County defendants and compared the predictions to what those people actually did over the next two years. Of those flagged as likely to commit a violent crime, roughly four in five did not. And Black defendants were roughly twice as likely as white defendants to be wrongly flagged as high risk.
By the time anyone could read those numbers, the numbers had already done their work. The scores were in the record. The people misclassified as high risk had already paid for it — in days of detention, in decisions whose effects compounded downstream. Nowhere in the chain of institutions that received the scores was there a mechanism to retrieve them. The analysis that proved the harm arrived years after the harm.
Transparency Is Not Accountability
A sentence recurs in almost every AI governance document of the last decade: automated decisions must be transparent. The sentence is true, and on its own it does very little. Transparency is a property of disclosure — what the system does, what data it uses, what logic it follows. Recourse is the other half of the exchange, the half that makes the first half matter: the capacity of the person on the receiving end to actually contest the decision, demand review, force a change, or stop the harm before it sets. Remove any one part of recourse — timely notice, a case-specific explanation, a reviewer who can actually disagree, suspension of the outcome, correction when wrong — and what remains can look like accountability without functioning as it.
COMPAS is the clean example of what happens when only the first half exists. After the ProPublica analysis, the vendor offered an account: the system satisfied a different, technically valid fairness criterion — predictive parity. The account was defensible on its own terms. For the defendants whose liberty had already turned on scores that entered the record before any challenge was possible, it was beside the point. The detentions did not become warranted because the vendor could defend a statistical property after the fact. The score had done its work; the explanation came later. That gap has a name: institutional narration — a system produces harm, the institution explains it afterward, and the explanation gets treated as accountability, even though nothing about the outcome can still be changed.
State v. Loomis: The Law Said This Was Enough
In State v. Loomis, decided by the Wisconsin Supreme Court in 2016, a defendant challenged a COMPAS score used at his sentencing — not on the ground that it was wrong, but on the ground that he could not see how it worked. He lost. The court let the score stand, attaching cautions to its use but leaving the proprietary logic intact. Loomis demonstrates the consequence directly: a person can be sentenced with the help of a number he is not permitted to examine, and the law will call that process adequate. Opacity by itself isn't illegal — trade secrets protect real commercial interests that predate AI. What's at issue is opacity joined to consequence, where a vendor's right to protect its model collides with a person's liberty.
Three Patterns Behind Every Recourse Failure
COMPAS is one instance of a structure that recurs across algorithmic harms. Three patterns explain why explanation so often fails to become recourse.
- Proprietary opacity — the model, training data, and threshold logic are protected as trade secrets, so defendant, counsel, and court alike cannot compel production of how the score was actually generated.
- Nominal human review — the law requires a human in the loop but rarely specifies what that human must actually do, so a reviewer exists on the org chart with neither the time nor the standing to disagree; the reviewer who routinely overrides the model triggers an audit, and the one who routinely accepts it hits productivity targets.
- Temporal asymmetry — the institution decides quickly, the affected person contests slowly. SCHUFA's loan-refusal case took years to reach Europe's highest court; COMPAS compresses the same gap into a tighter frame, with misclassifications happening weekly and surfaced in aggregate only years later.
The Recourse Audit: Three Things to Prove Before Deployment
Before deploying any consequential system — anything whose output touches a person's liberty, livelihood, housing, credit, health, custody, or immigration status — an institution should be able to prove three things while the system is still on the drawing board, not after the first complaint.
- The person must know in time — notice that an automated system is involved, in plain language, specific to their case, before the decision lands. Notice that arrives with the outcome is a receipt, not protection.
- The person must reach power — a human reviewer who is competent to judge the output, authorized to overturn it, and able to make that reversal actually bind what happens next. A reviewer who can object but never prevail is review in name only.
- The remedy must still matter — the process has to be able to pause the outcome while it's contested and reverse it in time to count. A detention served, a benefit cut off, a job filled — these are facts produced by speed, and a remedy that arrives after them is in the wrong order.
Why This Is a PIA Question
A Privacy Impact Assessment that only checks whether a scoring system's data is accurate misses exactly what COMPAS teaches: the score can be statistically defensible and still produce an unappealable harm, because no one built a channel for the person scored to reach it in time. The GDPR's Article 22, the AI Act's Article 86 explanation right, Quebec's Law 25 review right — all of them promise something close to recourse, but a right on paper is not the same as a remedy that arrives before the detention is served.
If your organization is deploying or evaluating any scoring or risk-assessment tool, the recourse audit above is the test to run before launch, not after a ProPublica-style investigation forces the question. This is the kind of assessment PIA Studio is being built to support at scale — one that asks not just whether a system works, but whether the person it scores can still be heard.