In 2014, the Dutch government gave itself the power to detect welfare fraud by machine. The system was called SyRI — Systeem Risico Indicatie, or System Risk Indication. It ran on fusion: records that separate arms of the state had collected for separate reasons — tax filings, benefit claims, employment data, housing records, debts — were pooled and passed through an undisclosed risk model. The model scored households in a targeted neighborhood and marked the ones it judged suspicious. Each mark became an investigation. Officials chose the neighborhoods; the model chose the households.
What set SyRI apart was not the scoring. It was the secrecy. The system was aimed at whole neighborhoods — in practice, low-income areas with large immigrant populations — and the people inside them were scored without their knowledge. For many residents, the first evidence that a machine had weighed their lives was an investigator at the door.
The Court Did Not Ask for a Better Version
A coalition of civil-society and human-rights groups took the government to court. Their claim was not that SyRI made too many mistakes. It was that no one could check it — not the people it judged, not the public, not a court — because it had been built to work in the dark. They did not ask the court to fix the system. They asked it to rule that a system built this way could not stand.
In February 2020, the District Court of The Hague agreed. It held that the legislation authorizing SyRI was incompatible with Article 8 of the European Convention on Human Rights — the right to respect for private and family life. The court's reasoning reached the architecture rather than the output: the concealment was not an unfortunate feature that better practice could repair. The opacity was not a flaw to be fixed. The opacity was the system.
Notice what the court did not do. It did not order the government to publish the model, tune it more carefully, or bolt on a better appeal. It did not call for more transparency or a fairer process. It stopped the system. The remedy was not a compliance order or an improvement plan. It was a prohibition.
The Mitigation Reflex
Most institutions do not know how to refuse a system. They know how to improve one. That is the mitigation reflex: the assumption that deployment is already settled, and that the only serious work left is to make the system less dangerous — tune the model, broaden the dataset, add review, publish guidance, build an appeal. All of those tools matter. But they belong to a world in which the system has already passed the first gate: the gate that asks whether it should exist at all.
Algorithmic governance has been slow to name that first gate plainly. It has often treated refusal as failure — a project canceled, a tool abandoned, a modernization delayed. But refusal is not failure when the system's premise is the problem. It is governance doing its first job.
Mitigation asks how to make a system safer. Refusal asks why the system should exist at all.
What Article 5 of the EU AI Act Already Says
The law has begun to create a category that algorithmic governance avoided for too long: systems that are not merely risky, but unacceptable. The clearest example is Article 5 of the EU AI Act. Its prohibited practices are not high-risk systems waiting for better safeguards — they cannot be rescued by an impact assessment, made acceptable by a human reviewer, or cured by a promise to monitor outcomes. They are outside the permission structure entirely.
Article 5 includes practices such as social scoring that produces unjustified or detrimental treatment, untargeted scraping of facial images to build recognition databases, certain uses of real-time remote biometric identification in public spaces for law enforcement, and criminal-risk prediction based solely on profiling or personality traits rather than objective facts linked to conduct. The Act entered into force in August 2024, and the prohibitions became applicable in February 2025. The list is narrow. That is the point: the law is not saying these systems need better deployment. It is saying they should not be deployed.
Other frameworks are less direct but point the same way. The GDPR limits solely automated decisions with significant effects. Brazil's LGPD gives the affected person a right to request review of solely automated decisions. Quebec's Law 25 requires organizations using exclusively automated decision-making to disclose that fact and give the affected person a path to submit observations to a human reviewer. None of these draw the same line as Article 5, but together they show a direction of travel — from a world where every system is presumed deployable with enough safeguards, toward one where some systems must justify their existence before safeguards are even discussed.
Refusal, Late and On Time
SyRI is the anchor because the refusal came from a court before the system could be normalized as ordinary administration. Other cases show what happens when institutions refuse too late, or only after the harm has become impossible to ignore. Australia's Robodebt scheme converted uncertainty into debt and placed the cost of disproving it on people with the least power to absorb the error; when the Royal Commission concluded in 2023 that the scheme had been neither fair nor lawful, the lesson was not that Robodebt needed better calibration — it was that the system should have been stopped before it became machinery.
Facial recognition produced a different form of refusal that arrived closer to on time. Between 2019 and 2021, several U.S. cities restricted or banned law-enforcement use of the technology; San Francisco moved first, in May 2019. In 2020, major vendors stepped back in different ways: IBM withdrew from general-purpose facial recognition, Amazon paused police use of Rekognition, and Microsoft said it would not sell the technology to police departments until federal rules existed. A city ban, a corporate exit, and a sales moratorium are different tools, but they carried the same institutional recognition — that the question was not how to make the technology fairer, but whether it should be used at all.
The Refusal Threshold
A system crosses the line past which the responsible decision is not to build when any one of the following conditions is present.
- Human legitimacy is non-delegable. Sentencing, child custody, asylum, deprivation of liberty, and end-of-life medical judgment may be informed by a system, but the authority itself cannot be transferred to a probabilistic mechanism.
- The harm is irreversible at human scale. A detention served, a benefit cut off at the moment of need, a family separated, or a wrongful arrest cannot be repaired by later explanation.
- The system depends on opacity. When a system can operate only because the people it affects cannot know how they were classified or how to challenge the result, opacity is not a flaw — it is the architecture.
- The risks fall on the classified while the benefits accrue to the institution. Efficiency for an agency, bank, court, or police department cannot justify a structure in which ordinary people carry the error, delay, stigma, and burden of proof.
- The premise feeds historical harm forward. When a system depends on records produced by over-policing, poverty surveillance, or prior exclusion, the premise itself may be disqualifying.
These are not factors to be balanced into a passing score. They are standards. One is enough. The decision not to build is also a decision — it deserves evidence, ownership, and a record.
Why This Belongs in a PIA
The pressure to deploy is easy to understand: deployment has sponsors, budgets, vendors, timelines, and promised savings. Refusal has almost none of that. A system that is never built produces no launch announcement. But the failure to refuse is not invisible — it appears later as debt notices, investigations, arrests, and public inquiries asking why no one stopped the system before it became ordinary administration.
A Privacy Impact Assessment that only asks how a system's use of personal data can be made safer will not catch this. Before an institution asks how to govern a system, it has to ask whether the system deserves to exist at all — and that question belongs in the PIA process itself, not as an afterthought once procurement is already underway. If your organization is evaluating a system that scores, sorts, or flags people at scale, the refusal threshold above is the test to run before the first line of code is written, and it's the same judgment PIA Studio is being built to support.