Starting around 2014, Amazon's engineering team built an automated recruiting tool with a simple brief: read a résumé, and tell a recruiter, on a roughly five-star scale, whether the candidate was worth interviewing. To learn that judgment, the system was fed résumés Amazon had received over a ten-year window, paired with the hiring outcomes attached to each one — a machine trained to predict, from the text of a résumé alone, the kind of candidate Amazon had previously chosen to hire.
By 2015, the team saw what the system had also learned. It was downgrading résumés containing the word "women's" — as in "women's chess club captain" — and ranking graduates of all-women's colleges lower than graduates of comparable coeducational schools. Amazon had never given the model a gender field. It inferred gender anyway, from phrases and institutions that had functioned, in ten years of hiring data, as indirect signals of the applicant's sex.
Editing Out the Words Didn't Fix It
Engineers tried editing the model to ignore the offending terms. It didn't work. The model had learned, across thousands of features, what kind of résumé tended to be associated with hiring success at Amazon — and removing one signal simply exposed the others. There was no clean way to tell the model to stop being right about Amazon's own history. In October 2018, Reuters reporter Jeffrey Dastin reported that the project had effectively been scrapped by early 2017. Amazon told Reuters the tool had never been used by recruiters to evaluate live candidates — it had been an experiment that, by the company's account, made no hires.
That distinction matters less than it appears to. What Amazon built and then walked away from was a working illustration of what happens when an organization asks a machine to learn from its own past, and the past is not a neutral record. The system had not been instructed to prefer men. It had been instructed to learn what success at Amazon looked like — and success at Amazon, over the prior decade, had mostly looked like men.
Why "Remove the Protected Fields" Doesn't Work
The standard fix for algorithmic hiring bias is to strip the résumé of protected characteristics — remove sex, race, age, national origin — on the theory that a scrubbed input produces a clean output. Amazon's case already shows why that fails: the model never had a gender field to begin with, and it found gender anyway.
A résumé is a dense record of a person's history, and a person's history is where their protected characteristics live. A degree from a women's college can stand in for gender. A foreign or unfamiliar name can signal perceived national origin. A postal code can carry the residue of race, wealth, and access. A multi-year employment gap can read as caregiving, illness, or incarceration. None of this has to be labeled for a model to learn it — the model simply learns which threads in the document correlate with the outcome it was trained to predict, and weights them. Remove the obvious proxy and the correlation redistributes itself across whatever proxies remain.
The Deeper Problem: What "Successful Hire" Actually Means
Auditing and cleaning the input data is necessary, but it's incomplete, because the harder problem sits in the label the model was trained to predict, not the résumé text it was trained on. "Successful hire" is not a fact sitting in the data — it's a judgment that Amazon's own managers applied to real people, shaped by the same institutional culture and blind spots the model is now being asked to learn from. A woman on a team that didn't believe she belonged might have received lower performance reviews not because she performed worse, but because she was seen differently. By the time that judgment becomes a training label, the asymmetry is already built into every line of the data — and no amount of cleaning the résumé features fixes a label that was contaminated at the source.
Where the Law Actually Lands
Four frameworks have started drawing the contours of accountability for automated hiring, with real differences in how far each one reaches. The EU AI Act classifies AI used to screen or evaluate job applicants as high-risk, requiring the vendor to document training data and design for human oversight, and the employer to run the system as specified, keep logs, and tell workers before it's used on them — sitting alongside the GDPR's Article 22 right not to be subject to a solely automated decision with significant effects. Brazil's LGPD Article 20 gives a person the right to request review of a decision made solely on automated processing, reaching automated hiring that crosses the "solely automated" threshold, though a dedicated enforcement line aimed specifically at hiring tools has not yet been publicly established. In the United States, the EEOC confirmed in 2023 that Title VII reaches algorithmic selection and that an employer can be liable for a vendor's tool under a disparate-impact theory — guidance the agency withdrew in 2025, though the underlying statute and the employer's exposure remain unchanged; city and state law fills in unevenly, with New York City's Local Law 144 requiring a bias audit and candidate notice. Canada answers least at the federal level, relying on PIPEDA, human-rights law, and Quebec's Law 25, which requires that a person subject to an exclusively automated decision be told it happened, given the principal factors behind it, and allowed to submit observations for review.
Read across all four, the laws increasingly agree that automated hiring is consequential enough to regulate. They agree far less on whether the rejected candidate can actually reach the notice, explanation, or review the statutes promise — and in hiring, that gap between what the law says a candidate is owed and what they can actually obtain is often the whole case.
Three Questions Before Deployment
Before an organization lets an algorithm screen résumés, it should be able to answer three questions in writing.
- What past outcome is the system being trained to reward? An algorithm doesn't learn "merit" in the abstract — it learns a target the institution chose, like hired, retained, or promoted. If that target was shaped by biased managers or uneven opportunity, the model will learn that history as if it were talent.
- Who does the system reject more often? That has to be measured before launch and checked again after deployment, across gender, race, disability, age, and other groups — a vendor's audit isn't enough unless the employer understands the method, the groups tested, and the test's limits.
- Who can reverse the machine? A score with no override isn't advice — it's the decision wearing a recruiter's name. The organization needs to know who can review a rejected candidate, what record they'll see, and how an error gets corrected.
Why This Is a PIA Question, Not Just an HR Question
An automated hiring tool that screens or scores candidates is exactly the kind of processing that triggers a Privacy Impact Assessment obligation under the frameworks this series has already covered — high-risk profiling under the GDPR, automated decision-making under LGPD Article 20, and the project-based triggers under Quebec's Law 25. A PIA done honestly before deployment is the document that forces an organization to name its training target, measure its rejection pattern by group, and assign a real reviewer — the same three questions Amazon's engineers eventually asked themselves, just asked before the system ships instead of after a Reuters investigation.
That is the throughline of this series: the gap between what a system does and what an institution lets it mean is where the harm actually happens, whether the system is scanning a face or scoring a résumé. If your organization is evaluating any AI tool that screens, ranks, or scores real candidates, those three questions are the starting discipline — and it's the same judgment PIA Studio is being built to support at scale.