I learned division when I was ten or eleven years old, in the fourth grade of a public school in Teresina, the capital of Piauí. I had been born in Campo Maior, a town a few hours away, but left before I could form any memory of it. Whatever I would learn of arithmetic, categorization, and the world, I would learn in the capital. Piauí has long stood among Brazil's poorest states. My school had no library, no laboratory, and no quiet expectation that children like us would one day enter a university. No one said this directly. They did not need to. Institutions often teach by what they withhold.
Division, as I learned it then, was simple. You took a quantity and broke it into equal parts. Thirty-six divided into groups of six became six groups. The numbers obeyed. The chalkboard, the notebook, and the answer key all agreed. And beneath the arithmetic, there was a deeper lesson: the world could be divided. Things could be counted, sorted, compared, and placed into groups assumed to be equal in nature.
I did not understand it at the time. I was not learning arithmetic alone. I was learning a cognitive gesture: the act of taking a messy, continuous world and breaking it into categories. Boys and girls. Black and white. Risky and safe. Approved and denied. Creditworthy and not. Recidivist and non-recidivist. This is the foundational gesture of every classification system. Before a machine can decide whether someone receives a loan, a job interview, a parole hearing, or an insurance premium, it must first do what I learned in that classroom in Teresina. It must break reality into bins.
My fourth-grade teacher was kind. I believe she did the best she could with what she had. She thought she was teaching me division. She was also teaching me the architecture of artificial intelligence. The same may be true of many engineers today. They think they are teaching machines to calculate. Often, they are teaching them to divide the world according to categories humans invented, institutions recorded, and data preserved.
When Learning Fails to Generalize
I learned the difference between recognizing a pattern and understanding a subject the hard way. That is why I can recognize the same failure now in a machine. In high school, mathematics and physics were the subjects I trusted; they came easily. On Brazil's university entrance exam, I answered fifty-nine of the sixty questions in the mathematics and physics section correctly. By every measure the system offered, I had mastered the material.
Then I began a computer science degree, met university mathematics, and failed Calculus I. Not struggled. Failed. The failure stayed with me because it contradicted the story my records had told. It took me years to understand what had happened. I had not learned mathematics as a language of proof. I had learned it as a route to an answer. The exam rewarded one motion: take the given data, choose the formula, calculate until a number appears. I had practiced that motion thousands of times. But Calculus asked for abstraction, generalization, reasoning that could hold beyond the cases I had already seen. What I had perfected did not transfer.
In machine learning, there is a name for this failure: overfitting. A system is overfit when it performs well on the examples it has already learned from but fails when it meets new cases that differ from its training. It has not learned the world. It has learned the exam. That was my problem. My score said I understood mathematics. What I understood was the test.
Years later, English taught me the same lesson in another language. My school records in Brazil certified, year after year, that I had studied it. On paper, I had passed. But those records had measured classroom English, not living English — exercises, not conversation; recognition, not use. The real test came during a trip to London. I remember standing near Abbey Road, in the city whose language I had supposedly studied for years, and realizing that English did not arrive as a sentence printed on a page. It arrived quickly, imperfectly, with accents, interruptions, jokes, noise, impatience, and no time to translate. That was when the training failed. My record said one thing. London said another.
A system learns the pattern in front of it. It performs brilliantly on material that resembles its training. Then the world changes, or the people before it do not resemble the people in its records — and the system fails while its score still suggests competence.
From Human Learning to Machine Learning
The gap between performance and understanding matters because machines learn in a way that can look strangely familiar. Think of a child learning what a dog is. At first, the child may know only the family dog — fur, four legs, a tail, a bark. Then the child sees another dog, smaller and louder, and still learns to call it dog. With enough examples, correction, and repetition, the child begins to form a category that can survive variation. But the learning is not perfect: a child may call a cat a dog because it has four legs and fur, or may refuse to call a tiny Chihuahua a dog because it doesn't match the first example closely enough.
Machine learning performs a narrower version of that gesture. It is shown examples. It extracts patterns. It uses those patterns to classify new cases. What the child receives as experience, the machine receives as training data. What the child receives as correction, the machine receives through a training method. What the child retains as a concept, the machine retains as a model. I would later spend years writing software and teaching programming, long before I entered law. That experience did not make the analogy less strange to me. It made it more precise: computers do not learn as children learn, but both systems depend on examples, correction, boundaries, and the categories someone gives them.
The comparison is imperfect, but useful, and it exposes a real limitation. A child can eventually question a lesson, resist an adult, or learn that a rule was wrong. A machine cannot. It has no memory of injustice, no embarrassment, no conscience, and no independent reason to distrust what it has been shown. If a model sees only certain kinds of successful applicants, safe borrowers, or compliant patients, it may learn the boundary of the category from that narrow world and treat everything outside it as abnormal.
This is why one word hides too much: the algorithm. People use it as if it named the whole system. It does not. Training data is what the system is shown. The training method is the procedure used to extract patterns from that data. The trained model is the pattern fixed into usable form. The prediction is the answer the model gives when a new case appears. Once those pieces are separated, the source of harm becomes harder to oversimplify. When an automated system produces an unjust result, blaming "the algorithm" is often too crude — the problem may be the data, the labels attached to it, a proxy standing in for something the institution never measured directly, the model's learned boundary, or the human being who treats the output as a verdict.
When the Model Gets It Right — and the Result Is Unjust
The easier story says the algorithm got it wrong: a faulty match, a mislabeled image, a bad threshold, a bug. If that's the problem, the solution remains comforting — better data, better testing, better math, a better version. But many of the hardest cases do not begin with malfunction. Sometimes the model gets it right, and the result is unjust. The system learned what it was allowed to learn. It reproduced patterns embedded in records made by human beings, inside institutions shaped by history, incentives, scarcity, prejudice, and habit. By the time the model begins to learn, the world has already been sorted for it.
That is why accuracy can become dangerous. A model can accurately reproduce an unequal distribution. It can accurately learn who was previously approved, hired, promoted, stopped, searched, treated, ignored, or denied. It can accurately inherit the past without understanding what made the past unjust.
The model itself makes no moral judgment. It does not know whether a pattern is fair, lawful, or decent — it knows only that the pattern was present often enough to matter. The machine does not author its values. It inherits ours.
Why a Lawyer Tells This Story
I did not need a machine-learning textbook to understand overfitting. I needed a failed Calculus exam and a confusing afternoon near Abbey Road. Both taught me the same lesson a model teaches an engineer who watches it fail on data it has never seen: a record of past performance is not the same thing as understanding, and a system trained on a narrow slice of the world will treat everything outside that slice as an anomaly, not as evidence that the training was too small.
That lesson is why a Privacy Impact Assessment for any AI system has to ask not just what data trained the model, but whose reality that data actually represents — and whose it leaves out. AI does not begin with code. It begins with the world we record, and that record is never as complete as it looks from inside the institution that made it.