Quick answer:
A hallucination is when an AI model states something false with complete confidence: an invented statistic, a citation to a paper that doesn’t exist, an API function that was never written. The output is fluent, plausible, and wrong, with no signal separating it from the model’s correct answers. It’s the central reliability problem of large language models, and the reason “the AI said so” is never sufficient evidence for anything.
What is a hallucination?
A language model is trained to predict plausible next tokens, and truth and plausibility usually travel together, because the training text mostly describes the real world. Hallucination is where they part ways: the model produces something that reads exactly like a fact without being one.
The tell is that there’s no lookup happening. The model has no database of facts it consults and no internal flag for “I’m making this up”. Knowledge is smeared across billions of numeric weights, and generating a real citation and a fake one feel identical from the inside. That’s why hallucinations arrive with the same confident tone as everything else, and why they’re so much more dangerous than an honest “I don’t know”.
What do hallucinations look like in the wild?
The canonical cautionary tale is Mata v. Avianca (2023): a New York lawyer submitted a brief containing 6 court cases suggested by ChatGPT. None existed. The judge fined the lawyers $5,000 and the story made every front page, permanently attaching “made-up court cases” to the word hallucination.
The everyday versions are quieter: a code assistant importing a library function that doesn’t exist, a chatbot inventing a refund policy (Air Canada was held liable in 2024 for a policy its support bot made up), a research summary with one fabricated number among 9 real ones. That last shape is the worst, because the 9 real ones buy credibility for the fake.
Why can’t they just fix it?
Because the mechanism that hallucinates is the same one that does everything else. Next-token prediction is the whole engine; you can’t remove the confabulation without removing the fluency. Asked for an answer it doesn’t have, the model does what its training rewarded: produce the most plausible-sounding continuation.
What vendors can do, and have, is push the rate down: better training data, reinforcement learning that rewards refusals over fabrications, models that cite sources. Rates dropped substantially between 2023 and now, especially for grounded tasks. But the floor isn’t zero, and a system that’s right 98% of the time with no marker on the 2% remains a system you have to verify.
How do builders reduce hallucinations?
- Ground the model. RAG hands the model real documents and instructs it to answer from them. This converts “recall from weights” into “summarize what’s in front of you”, a far easier task, though the model can still misread what it was handed.
- Demand citations. When every claim must point at a retrieved source, fabrication gets harder and checking gets possible.
- Give an exit. Prompting matters: models explicitly permitted to say “not in the provided material” fabricate less than models cornered into answering.
- Constrain the task. Classification, extraction, and structured output hallucinate less than open-ended generation.
- Evaluate continuously. Serious teams measure groundedness with eval suites and treat regressions like failing tests, the same discipline this site would call observability, applied to model output.
Is hallucination all downside?
Where the same mechanism helps
Generation is confabulation, aimed well
Brainstorming, drafting, naming, and fiction all want the model to produce things that don’t exist yet. The property only becomes a bug when the task demanded facts.
Filling gaps is often the job
“Write the missing test cases”, “suggest what this abstract might conclude”: useful work built on plausible invention, clearly labeled as such.
Where it does real damage
Fabricated specifics in factual clothing
Invented citations, statistics, case law, and API functions, delivered in the register of fact. This is the shape that fines lawyers and ships broken code.
Errors that compound downstream
One hallucinated number pasted into a report becomes a source for the next report. Unverified AI output entering a data pipeline is contamination with a long half-life.
Confidence that defeats skepticism
The fluent, assured tone suppresses exactly the doubt that would catch the error. Humans calibrate trust on delivery, and delivery is the one thing the model always nails.
Trust erosion cuts both ways
After the first caught fabrication, users start re-verifying everything, and the assistant’s time savings quietly evaporate. Reliability was the product; the failure taxes every future answer.
How should a working professional treat this?
Adopt the journalist’s rule: verify anything that would embarrass you if wrong. Names, numbers, quotes, citations, legal and medical claims, function signatures. Verification is cheap next to retraction.
And calibrate by task, because that’s what the failure statistics actually reward. Asking a model to rewrite your paragraph: hallucination risk is near zero, edit freely. Asking it to summarize a document you provided: low risk, skim-check. Asking it for facts from memory, especially obscure ones: that’s the danger zone, treat the output as a lead, and check the source before you print it. We hold ourselves to that last rule for every number in this directory, for exactly this reason.