In October 2013, the State of Michigan turned its unemployment system over to a machine. The Michigan Integrated Data Automated System — MiDAS — was built to find benefit fraud and to do it without people. It compared a claimant's records, looked for discrepancies, and where it found one, it decided. A mismatch became a question. A question left unanswered for ten days became a finding that the claimant had knowingly lied to collect money they were not owed.
The finding carried consequences the system also applied on its own: penalties that could reach four times the amount in question, wage garnishment, seizure of tax refunds. For tens of thousands of residents — more than forty thousand, by later counts — the first sign that anything had gone wrong was money leaving their accounts. When the state's auditor general later examined a sample of the fraud findings the system had produced without any human review, about ninety-three percent were wrong. MiDAS had not detected fraud. It had manufactured it, at volume, and then collected on it.
Michigan stopped using MiDAS to decide fraud automatically in 2015, under federal pressure and a lawsuit. Years of litigation followed, and the state eventually settled for twenty million dollars, with final court approval in January 2024. But the people garnished in 2014 did not have a class action. They had a portal, a deadline, and a machine that had already decided.
Reasons Are Not Power
For a decade, the reform conversation in AI governance has been about opening the box: make the system transparent, make it explainable, tell the affected person what data was used and what logic applied. Those are real gains. But transparency and explanation share a ceiling, and it's the ceiling MiDAS exposes. Both end at understanding. A score can be transparent in its inputs, explainable in its logic, and still land on a person who cannot move it an inch.
Many of the MiDAS letters did, in a sense, explain themselves — they cited a discrepancy, named an amount, pointed to a statute. A person could read the letter, understand the accusation completely, and still have nowhere to take it. The explanation was never the missing piece. The missing piece was a human with the authority to look at the discrepancy, agree that it was a data-matching artifact, and erase the debt.
A right to explanation without a right to contest is a courtesy. A right to contest without a reviewer who can change the outcome is theater.
A Human Reviewed It. A Human Could Have Changed It.
Two sentences sit at the center of this problem, and they sound almost identical. "A human reviewed the decision" is what many deployments offer as oversight — somewhere in the workflow there is a person who sees the output and clicks approve or deny. But that person is often not independent of the system, not trained to disagree with it, and measured on how many cases they clear rather than on the quality of their objections. Presence is not the same as power.
"A human could have changed the decision" describes something an institution has to build and pay for: a reviewer who can see the inputs, the record, and the basis for the output; who has the time to look; who has the authority to overturn, with the institution bound when they do; and who is protected when they say no. MiDAS removed the human outright. Most systems are subtler — they keep the human and remove the power, which photographs the same in a compliance file and means nothing to the person on the other end.
What the EU AI Act Already Requires
The law has begun to name this standard in operational terms. Article 14 of the EU AI Act requires high-risk systems to be designed so that real people can oversee them while they run — not people standing near the system, but people able to understand its limits, interpret its outputs, resist the pull of automation bias, and disregard, override, reverse, or stop the system when intervention is required. A reviewer who looks at an output and has no real ability to change it is not oversight. That reviewer is a checkpoint the machine has already cleared.
Article 86 adds a narrower companion right: in covered high-risk deployments, an affected person can require from the deployer a clear and meaningful explanation of the system's role and of the main elements of the decision. That matters, but it also exposes the limit — explanation opens the file; contestability asks whether anyone can still move the decision. The law can name the right. The system still has to build the door.
Where the Door Is Painted On
Australia's Robodebt scheme shows the most common shape of failed contestability: not silence, and not outright refusal, but a mechanism that exists on paper and collapses the moment a real person, with real constraints, tries to use it. The scheme raised welfare debts by averaging income and reversed the ordinary order of proof — the government asserted, and the citizen had to disprove. To contest a debt, a person needed records reaching back years; employers had closed, records were gone, and the online portal that received challenges did not help anyone reconstruct them. It waited.
The Royal Commission that reported in July 2023 found this was not an accident of implementation. The contest process had been built around the assumption that the debt was valid. There was a portal, there was a process — it looked like a door. A person could stand in front of it, follow every instruction, and still not get through.
Nine Standards a Contestable System Has to Meet
These are not questions to put to a system. They are properties a system either has or lacks.
- Notice arrives before the outcome hardens — while there is still time to affect it, not with the garnishment.
- The explanation is specific enough to act on — the reason this decision fell on this person, not a general description of the model.
- The path back in is not harder than the path that produced the decision — if the machine decided in milliseconds, the person shouldn't need to retain counsel and assemble years of records.
- A reviewer exists in operational reality — a specific human with hours and a caseload that permits actual attention, not a role described in a policy document.
- The reviewer has authority, time, independence, and protection — authority to overturn, time to evaluate, independence from metrics that reward agreeing with the machine, and protection from retaliation for disagreeing.
- The outcome pauses where harm would otherwise become irreversible — collection before contestation turns a reversible error into a permanent one.
- The record is preserved and reconstructable — a decision that cannot be reconstructed cannot be contested.
- Correction travels as far as the error traveled — fixing the source while leaving the copies standing, in a credit bureau or partner agency, is not correction the affected person will feel.
- Successful challenges change the system — repeated successful challenges are evidence of a defect, and must trigger a change in the model, the policy, or the deployment itself.
Why This Is Also a PIA Question
A contestable system is not a system that never fails. It is a system that gives people power when it does. None of the nine standards installs itself — each one costs an institution something it would rather keep, from a reviewer's salary to the delay of a pause. Left to its own incentives, a deployment will build the painted door, because the painted door is cheaper.
A Privacy Impact Assessment that stops at asking whether an automated system's data use is lawful will miss this entirely. The harder question — whether the person the system decides about can actually reach a human who can change the outcome — belongs in the assessment itself, before the system is switched on. If your organization is deploying any automated decision system with consequential effects, the nine standards above are the audit to run before launch, and it's the same discipline PIA Studio is being built to support at scale.