On October 30, 2020, I walked into a bank branch in Teresina, Brazil, to collect a piece of paper the law already said was mine. It was a discharge document — a termo de quitação — the letter a lender issues once a loan is paid off, confirming the debt is closed and the property lien can be lifted. Eight years earlier I had financed an apartment through CEF, a major federally owned Brazilian bank. That September, I had paid the loan off early. Brazilian law gave the bank thirty days to issue the discharge.
So I went back to the branch and asked for it. The employee checked the system and told me the document could not be issued — because I had an open lawsuit against the bank over unrelated charges, and because I carried unrelated balances there, an overdraft line and a credit card. Both facts were true. Neither was relevant. The statute conditions the discharge on one thing only: that the debt is paid. What the branch had was a screen that showed an open case and an open balance — and the screen, in effect, said no.
A True Fact Is Not a Legitimate Decision
Until the lien was lifted, I could not complete a sale I intended. To restore a distinction the law had already drawn in plain words, I eventually filed federal case no. 1003560-58.2021.4.01.4000 against CEF. The bank did not even contest it. Years later, a federal court ruled what the statute had said all along — the debt was paid, the discharge was owed — and applied the penalty Brazilian law prescribes for exactly this failure: half a percent a month on the value of the contract, a penalty that by then had grown beyond the apartment's original purchase price.
The data was not wrong. I had sued the bank, and I did owe money on a card. The data was true. The decision was unlawful. The bank's system knew I had a case against it. It did not know why — and the employee facing me had no authority to ask, and no power to act on the answer if she had. A true data point is not a legitimate decision. A record is raw material; a decision is what an institution chooses to do with it. The first is a matter of fact, and machines are excellent at it. The second is a matter of authority and law, and no system, however accurate, can supply it.
Governance Is Not a Document
The distinction this case turns on is the one most AI governance conversations skip: a true data point is not a legitimate decision. A screen that displays open litigation and pending balances is a record. The refusal to issue a discharge is a decision. Between the two there is supposed to be a person exercising judgment — someone who can look at a true fact and ask whether it's a lawful fact for this purpose. In that branch, the distance between the record and the decision had collapsed to nothing.
Governance is not a document. Governance is the power to interrupt a decision before it becomes harm. Most of what gets written about AI governance describes policies, frameworks, and audit reports — those artifacts may be evidence that governance exists, but they aren't governance itself. The real test is not whether an organization has a policy. It's whether, when the system is wrong, someone can say so and have it matter. A human pressing "confirm" is complying, not overseeing. Oversight begins only where disagreement is possible — and survivable.
What the Frameworks Actually Require
The major automated-decision frameworks, written on different continents out of different legal traditions, converge on this point. NIST's AI Risk Management Framework (2023) calls for "effective challenge" — people positioned, trained, and empowered to question an automated output and change it, paired with a warning against "techno-solutionism," the habit of treating a system's answer as the end of inquiry. The EU's GDPR Article 22 gives a person the right not to be subject to a decision based solely on automated processing with significant effects, with safeguards including human intervention. The AI Act converts that right into an organizational duty: Article 14 requires high-risk systems to be built for human oversight; Article 26 requires deployers to assign that oversight to people who are competent, trained, and given the authority to intervene; Article 27 requires a fundamental rights impact assessment before certain high-risk systems go live.
Brazil's LGPD gives individuals the right to request review of solely automated decisions — but when the law was amended in 2019, the requirement that the review be performed by a natural person was removed, leaving a review right without a guaranteed human reviewer. Quebec's Law 25 closes that gap more directly: when a decision is based exclusively on automated processing, the organization must disclose that fact and let the affected person submit observations to someone positioned to actually review — and change — the outcome.
The Same Wall, at Every Scale
One person at a bank counter is a small failure. A welfare agency doing the same thing to hundreds of thousands of people is a catastrophe — same structure, different scale. Between 2015 and 2019, Australia's automated Robodebt program compared reported annual income to fortnightly income and treated any mismatch as a debt, pushing the burden of disproving the number onto recipients who had no real channel to contest it before the harm landed; a 2023 Royal Commission found the scheme neither fair nor lawful. In the Netherlands, the tax authority's fraud-risk system flagged families receiving childcare benefits, and a flag became a verdict — benefits frozen, tens of thousands of euros demanded back, dual nationality and low income treated as markers of suspicion. The scandal, known as the toeslagenaffaire, brought down the Dutch government in January 2021.
In every one of these cases the structure repeats: the system says no, the person facing the customer cannot explain why in any meaningful way and could not change it if they could, and the affected person can object in theory but not through any channel likely to reverse the decision before the harm lands. That's the wall. It's the same wall whether it stands between a borrower and a discharge document, or a family and a tax authority.
Five Gates a Record Has to Pass Through
Before a true record becomes an institutional act, it should pass through five checks — and any organization deploying an automated or semi-automated decision system should be able to name who staffs each one.
- Relevance — does this fact actually matter for this decision? A lawsuit can be real and still irrelevant to whether a paid debt gets discharged.
- Purpose — is the data being used for the reason it was collected? An overdraft balance exists to manage an overdraft, not to block an unrelated discharge.
- Authority — who is allowed to turn this record into an institutional act? If the answer is "the system," there is no gate.
- Contestability — can the affected person challenge the decision through a channel that can actually reverse it in time to matter, not a complaint box that changes nothing?
- Accountability — who answers when the data is true but the decision is unlawful, irrelevant, or unfair?
Why This Belongs Next to a PIA
This is exactly the gap a Privacy Impact Assessment done honestly is supposed to catch before deployment: not just whether a system's data is accurate, but who has the authority to override it, and through what channel. A PIA that only asks about data quality misses the failure this chapter documents — the harm here didn't come from a wrong fact, it came from an institution that never built anyone able to say the fact didn't apply.
The statutory penalty in my case is, in its own strange way, a piece of legislative honesty — it puts a number on the missing gate. Most institutions don't get that clean a signal. If your organization is deploying any system whose output can become an automatic refusal — a discharge, a benefit, a credit decision — the five gates above are the starting discipline for asking whether anyone inside the institution actually has the authority to say otherwise, and it's the same judgment PIA Studio is being built to support at scale.