AI When AI gets it wrong: hallucinations, bias and limits
AI bias: where it comes from and where it shows up
How bias enters an AI system through data, labels and objectives, the places it surfaces in real decisions, and the questions that expose it before it costs somebody something.
The short answer
- AI bias is a systematic error that falls harder on one group, and it enters through three doors: the data collected, the labels applied to it, and the number the training run was told to maximize.
- Deleting race, sex or age from the input does not make a system blind, because names, addresses, schools, employment gaps and writing style rebuild the same signal.
- An overall accuracy figure can hide near total failure for a small group, which is why any useful evaluation reports results broken down by group.
- Feedback loops make it compound: a system that shapes the outcomes it is later trained on will look like it is getting more accurate while getting narrower.
- The practical defenses are ordinary questions about what was predicted, who was in the data, how performance splits by group, and how a person appeals.
Bias in an AI system means a systematic error that falls unevenly on some group of people. It is not the system having opinions. It is the system being measurably worse, or measurably harsher, for one set of people than another, in a way that repeats every time it runs. Nobody has to be prejudiced for this to happen. A model learns the patterns in what it was shown, and if the world that produced that material treated people differently, the pattern is right there in the data waiting to be learned.
There are three doors it comes in through: the data, the labels, and the objective. Almost every real case traces back to one of them, and knowing which one changes what you can do about it.
The three entry points
The data
A model can only generalize from what it saw, and training sets get gathered where collection is cheap rather than where the population is. Published audits of commercial face analysis systems found much higher error rates on darker skinned faces, with the worst results for darker skinned women, and the training material was a large part of the cause. The same mechanism produces voice systems that struggle with regional, second language and older speakers.
Historical records add a second problem. A lending dataset does not record who could have repaid a loan. It records who was offered one and what followed, so it documents past decisions as much as past outcomes, and a model trained on it learns the decisions. Where training data actually comes from covers how these collections get assembled, and representativeness is rarely the criterion used.
The labels
Supervised systems learn from examples that a human marked up, so the marking is part of the teaching. Three things go wrong there. The definition may be a convenient stand in for the thing you care about, such as scoring a "good hire" as anyone who stayed two years. The guidelines may be ambiguous, so annotators fill the gap with their own judgment. And the annotator pool may be narrow, which is how text written in one dialect ends up flagged as rude more often than the same content in another. Preference ratings used in the tuning stage of a chatbot have the same property: they encode the tastes of the people who did the rating.
The objective
The third door is the forgotten one. A training run optimizes a single number, and that number decides who gets sacrificed. Maximize overall accuracy and the optimizer will trade away performance on a small group to gain a fraction of a point on a large one, because that is the better score. Maximize engagement and you get whatever holds attention. The objective is a choice, usually made for engineering convenience rather than for fairness.
Why deleting the sensitive field does not work
The standard first attempt is to remove race, sex, age and zip code from the input and declare the system blind. It does not work, because those attributes are reconstructable from everything else. Statisticians call the leftovers proxies.
A gap in employment, a named school, a club membership, a first name, a graduation year or a home address each correlate with protected characteristics strongly enough for a model to rebuild the signal without being told it. Purchase history, device type and the phrasing of your writing do the same job. The model is not hunting for the attribute. It is finding whatever predicts the label, and if the label came from a process that treated groups differently, the proxies are the most predictive features available.
This is harder to spot in a neural network than in a scorecard, because nothing is written down as a rule. As what is actually inside a model explains, the learned pattern lives distributed across millions of numeric weights, so you cannot open the file and read a line that says which group is penalized. You can only test the behavior from the outside.
How a 95 percent accurate system fails a whole group
Here is the arithmetic that hides most real failures. It uses invented round numbers to show the shape of the problem, not measurements of any real system.
Suppose a screening tool sees applicants who are 90 percent from group A and 10 percent from group B. Suppose it is right 98 percent of the time for group A and 68 percent of the time for group B. Overall accuracy is 0.9 times 98 plus 0.1 times 68, which is 95 percent. The vendor sheet says 95 percent accurate. Nearly a third of the people in group B get the wrong decision, and no single number in the report shows it.
That is why any evaluation worth reading is broken out by group, and why an aggregate score says almost nothing about whether a system is safe to deploy. It is also why bias survives testing so often: the test was passed.
One honest complication. There is no single definition of fairness to test against. Equal error rates across groups, equal selection rates, and equal accuracy of the scores themselves are different requirements, and when the underlying base rates differ they cannot all be satisfied at once. This is a mathematical result, not a political position. Anyone selling a system that is simply "fair" has chosen one definition and not told you which.
Where it shows up in practice
The recurring cases in the research and audit literature follow a small number of patterns.
| System | Which door it came in through | What it looks like to a person |
|---|---|---|
| Face detection and analysis | Data: narrow range of skin tones and lighting | Higher error rates for darker skinned faces, worst for darker skinned women |
| Speech recognition and voice assistants | Data: accents and dialects underrepresented | Repeated failures to transcribe, blamed on the speaker |
| Resume screening | Labels: trained on who was previously hired | Candidates filtered on proxies such as school, gap years or phrasing |
| Health risk scoring | Objective: predicts cost of care rather than severity of illness | Patients who historically received less care are scored as less sick |
| Content moderation | Labels: annotators unfamiliar with a dialect or community | Ordinary speech removed as abusive at a higher rate |
| Image generation | Data plus captions: occupations skewed in the source material | A single prompt returns a narrow range of people for some jobs |
Hiring is the one most readers will actually meet, from either side of the desk, and it is worth reading what screening software really does with an application before assuming the filter is a person. If you are choosing tools for a team rather than being filtered by one, the obligations you want written into the agreement belong in the clauses to check in an AI vendor contract, because performance broken down by group is something you have to ask for.
Feedback loops make it worse over time
A one off error is a problem. An error that feeds its own training data is a spiral. If a deployment sends more scrutiny to the places it already flagged, it generates more records there, and the next model reads those records as confirmation. If a hiring model selects a narrower pool and those hires become next year's training examples, the narrowing compounds. If a recommendation system shows people what people like them clicked, the categories it invented harden into fact.
These loops are hard to see because the system looks like it is improving. Its predictions match the outcomes better each cycle, because it is helping to produce the outcomes. Accuracy measured against your own influence is not evidence about the world.
The questions worth asking before you trust a decision
You do not need to audit a model to interrogate one. Any organization that has done the work can answer these, and the silence is informative when they cannot.
- What exactly was the model trained to predict, and is that the thing you actually care about, or a convenient stand in for it?
- Who is in the training data, and who is missing?
- Is performance reported separately for each group that matters, or only as one overall figure?
- Which features could act as proxies for a protected characteristic, and was that tested rather than assumed away?
- Does a human review the decision, and does that person have the information and the authority to overturn it?
- How does a person appeal, and does anyone track how often appeals succeed?
The law is moving, unevenly. In the EU, the AI Act puts uses such as employment, credit, education and essential services into a high risk tier that carries duties on data governance, documentation, logging and human oversight, described in what counts as high risk and what it requires. In the US there is no single federal statute. Existing discrimination, credit and employment law applies to automated decisions, regulators have said so plainly, and some states and cities now require bias audits or disclosure for hiring tools. This is general information rather than advice for a specific situation, and the detail depends on where you are.
If you are introducing any of this at work, put the group by group testing, the human review step and the appeal route in writing before launch rather than after the first complaint. An AI policy people will actually follow is the place those commitments survive a change of staff.
Common questions
Can you just fix bias by adding more diverse training data?
It helps and it is rarely sufficient. Better sampling fixes bias that came in through the data door, but it does nothing about labels that encoded past decisions or an objective that trades small groups away. Data collection also has limits of its own, since the groups hardest to sample are usually the ones already underserved by whatever produced the records.
Is a human reviewer enough to catch a biased decision?
Only if the reviewer can see the reasoning, has time, and is genuinely able to overrule the output. In practice people tend to defer to a score, especially when overturning it takes paperwork and agreeing with it takes none. Human oversight that exists on the org chart but not in the workflow is the most common version of this failure.
Are chatbots biased in the same way as scoring systems?
The mechanism is the same, the visible symptoms differ. Instead of a wrong decision you get skewed defaults: occupations assumed to belong to one gender, names treated as more or less professional, dialects corrected toward one standard. It matters most when a chatbot is quietly used to rank, screen or summarize people rather than to draft text.
How can I tell whether an automated decision affected me?
Ask directly, in writing, and ask what information was used. In the EU and the UK, data protection law gives you a route to ask about automated decisions that significantly affect you and to request human involvement. In the US it depends on sector and state, though credit decisions in particular carry a long standing right to be told the main reasons for a refusal.