AI When AI gets it wrong: hallucinations, bias and limits
Made up sources: why AI invents books, cases and studies
Why chatbots produce citations that look perfect but do not exist, which fields it happens in most, and a fast routine for verifying any reference you are given.
The short answer
- A fabricated citation looks right because the model learned citation format far more thoroughly than it learned any individual paper.
- Invented references are not all or nothing; the most common failure is a real paper attached to a claim it never made.
- Verify in this order: does the title exist, does the identifier resolve, did those authors write it, and does it actually say what you were told.
- Treat quoted sentences and page numbers as unverified by default, even when the source turns out to be real.
- Courts in several countries have fined or sanctioned lawyers who filed documents citing cases that never existed, which is the clearest warning available.
Ask a chatbot for sources and you will often get something beautiful: three references with plausible authors, a journal you have heard of, a volume and issue number, a page range and a DOI. Two of them may not exist. The reason they look so convincing is that the model learned the grammar of citation far more thoroughly than it learned the contents of any particular paper. Format is everywhere in the text these models are trained on and highly regular. Individual papers are rare, and most appear only as a title in a reference list, never as a summary you could check.
So when you ask for evidence, the model does what it always does: it produces the most likely looking text. In this case the most likely looking text is a well formed reference. Whether the work behind it exists is not a question the system is equipped to ask. Why AI makes things up covers the general mechanism; citations are the sharpest example of it because the surface detail is so persuasive.
Five ways a citation can be wrong
Treating this as a yes or no question, real or fake, will cause you to miss most of the failures. There is a spectrum.
| Failure | What you see | How you catch it |
|---|---|---|
| Fully invented | Title, authors, journal and year all fictional | Exact title search returns nothing |
| Frankenstein reference | Real authors, real journal, title stitched from other work | Authors exist but never published that title |
| Wrong attribution | A real paper credited to the wrong people or year | Record loads but details do not match |
| Right source, wrong claim | A genuine paper that does not say what you were told | You have to open it and read |
| Invented quotation | Real book or judgment, sentence that is not in it | Search the exact phrase inside the text |
The middle three are the dangerous ones, because a quick search partly confirms them and your checking instinct switches off. A reference that half checks out feels more trustworthy than one that fully checks out, which is a strange trick your attention plays on you, and it pairs badly with the even tone an AI keeps when it is wrong.
The invented quotation deserves particular caution. Quotation shaped text is easy to generate: the model knows the author's register, the topic and the sentence rhythm. What it does not have is the sentence. If you are about to put quotation marks around something in an essay, a brief, a board paper or a post, that string has to be found in the actual text before it goes out.
Where it bites hardest
Law is the field with the loudest examples. Judges in the United States, the United Kingdom and several other jurisdictions have sanctioned lawyers who filed submissions citing decisions that did not exist, sometimes with fines, referrals to professional bodies and public reprimands. The pattern in the reported cases is consistent: the drafter asked a chatbot for supporting authority, the output looked exactly like case law, and nobody pulled the reports. Case citations are unusually easy to fake convincingly because their format is so rigid.
Academia is next. Students submit reference lists in which a quarter of the entries cannot be found, and the marker notices, because markers search titles. Medicine and nutrition follow, where an invented trial can travel a long way before anyone checks it. Local journalism and genealogy have the same profile: lots of specific, sparsely documented material.
The common factor is not the subject. It is that the reader is unlikely to open the source and the writer is under time pressure. That describes the exact circumstances in which people reach for a chatbot in the first place.
A four step verification routine
- Search the exact title in quotation marks, in a general search engine and in a scholarly index or library catalog. Zero results for an exact title is close to conclusive. Be aware that a search engine may now show you an AI written answer above the results that confidently describes the nonexistent paper, so look for an actual record from a publisher, library or repository.
- Resolve the identifier. Paste the DOI into a DOI resolver, or the ISBN into a library catalog, or the case citation into a court or legal database. A DOI either loads a record or it does not, which makes this the fastest true or false test available.
- Check the authorship independently. Find the author's institutional page or publication list and confirm that this title appears. Two real researchers in the same field can easily be joined to a title neither wrote.
- Open it and read the relevant part. This is the step everyone skips and the one that catches the "right source, wrong claim" failure. Look at the abstract and the specific section, check the sample size and the population if it is a study, and confirm the finding is stated rather than inferred. Treat page numbers as unverified unless you have the paginated version in front of you.
Four steps sounds heavy. In practice, steps one and two take about twenty seconds per reference and dispose of most problems. Step four is the one worth your time, and you only need it for claims you are going to rely on.
What retrieval connected tools change
A chatbot wired to live search, or a research mode that browses and then writes, is meaningfully better here. It is quoting from pages it actually fetched, and it gives you links to follow. Research assistants built on retrieval, described in RAG or fine tuning, exist largely because grounding answers in retrieved documents cuts fabrication.
Three gaps remain. The system may retrieve a weak source, a preprint, a press release or a blog summary, and present it with the same authority as a peer reviewed paper, so judging whether a source is worth believing stays your job. It may retrieve one source and then pad the reference list from memory, leaving you with a mix. And the summarizing layer can still drift: the page says "associated with," the answer says "causes."
So the rule changes shape rather than disappearing. With an unconnected chatbot, assume every reference is unverified. With a connected one, assume the links are real and the characterization is unverified.
What to check first
Before you send anything containing AI supplied references, run the cheapest test on every entry: exact title search plus identifier resolution. Then pick the two or three citations your argument genuinely rests on and open those properly. If you are writing for study or for work, the routine in how to fact check an AI answer in five minutes generalizes this to claims without citations, and using AI for studying covers where the line sits between help and academic misconduct. One failed reference is a reason to check all of them, not just to delete that one.
Common questions
Why does the DOI look real?
A DOI is a short structured string, and the model has seen millions of them, so it can generate something with the correct shape and a plausible registrant prefix. Shape is not registration. The only test that counts is pasting it into a resolver and seeing whether a record loads.
The chatbot gave me links and they opened. Is that enough?
Not quite. Check that the page you land on is the work described and not a search results page, a login wall or a different article with a similar title. Then check that the source supports the specific claim, because a real link attached to a wrong summary is the most common version of this problem.
Can I ask the AI to verify its own citations?
It will usually comply and it will usually confirm them, including the fake ones, because agreeing is a likely continuation of the conversation. Verification has to happen outside the chat, in a library catalog, a journal site, a court database or a bookshop listing.