Of the six questions in the PIA Quick Check, one trips more often than the rest for teams building or buying AI tools: will an algorithm or automated system make decisions that affect people's lives? If the honest answer is yes, you're almost certainly looking at high-risk processing that needs a full Privacy Impact Assessment — not because AI is inherently dangerous, but because automated decision-making is exactly the category regulators worldwide have singled out as opaque, hard to challenge, and prone to reproducing bias at scale.
Why Automation Gets Its Own Trigger
The GDPR's Article 22 restricts solely automated decisions that create legal or similarly significant effects on a person — a narrow legal test, but one built around a broader concern that shows up across every major privacy framework: automation can be opaque, biased, and hard to challenge in ways a human decision-maker, whatever their flaws, usually isn't. That's why signals of risk under this question aren't limited to the word "AI" — they include any system that scores people, ranks them, or determines their access to jobs, credit, health, or education.
Four Examples, Four Different Risk Levels
The same underlying question — does an algorithm decide something that matters — can land in very different places depending on what's actually at stake.
- A credit-scoring algorithm that approves or denies loans is high risk, since it directly impacts financial rights — the kind of consequential, hard-to-reverse decision Article 22 was written for.
- An AI recruiting tool that filters résumés or ranks candidates is high risk, because it shapes access to employment opportunities — and, as the Amazon hiring case shows, can encode discrimination the organization never intended and may not even be able to see.
- A social media feed that uses profiling to deliver ads can be high risk when it targets vulnerable groups, like children or voters, but lower risk if it only suggests general content without targeting sensitive traits — the line is targeting, not personalization itself.
- A fitness app recommending workout routines is usually low risk, but the moment it starts offering medical or diagnostic advice, it crosses into high risk — the same technology, a different claim about what it's deciding.
An online exam platform that automatically flags students as cheaters is worth naming on its own, because it's a category that keeps recurring in privacy complaints: it's high risk precisely because the output — a cheating flag — can determine academic success or failure, often without the student ever seeing how the flag was generated.
The Rule of Thumb
If a system decides for or about people without meaningful human review, treat it as high risk. When in doubt, conduct a full Privacy Impact Assessment to document fairness, safeguards, and explainability — not as a formality, but because those three things are exactly what a rejected candidate, a denied applicant, or a flagged student will ask about if they ever get the chance to push back.
The Bridge to Algorithmic Bias
This is where a Privacy Impact Assessment and an algorithmic bias audit start asking the same question from different directions. A PIA asks whether a system's use of personal data creates a privacy risk; a bias audit asks whether a system's output distributes unfairly across groups. Where the two meet is automated decision-making — Amazon's hiring algorithm and Robert Williams's facial recognition case are both, at bottom, PIA-trigger stories: automated systems making consequential decisions about real people, deployed without the documentation that would have forced someone to name the training target, measure the rejection pattern, or assign a real human reviewer before the system shipped.
If you're evaluating whether an AI tool your organization is building or buying needs a full PIA, the test from this chapter is simple to state and hard to fake: does the system decide for or about people, and is there meaningful human review before that decision becomes final? If the answer to the first is yes and the second is no, you already have your answer. This is the third article in a series drawn from Privacy Impact Assessment Basics — and it's exactly the judgment PIA Studio is being built to support at scale.