beginner25 minby DevFox

Why Models Hallucinate

The mechanism behind confident invention, why asking a model to be accurate does not make it accurate, and the three things that genuinely reduce it: grounding, permission to refuse, and verifiable citations.

You are adding retries to an API client and ask the model for the right option name. Back come four tidy lines with a comment explaining the backoff, built around retry_backoff_seconds=2. You paste it in and get TypeError: unexpected keyword argument. The library has retry_delay, and has never had retry_backoff_seconds.

Nothing went wrong in the usual sense. No error, no warning, no hesitation — the model produced the most plausible-looking answer available, and plausible is not the same as true. By the end of this lesson you will know why that happens, which questions provoke it most reliably, why "be accurate" in a prompt does so little, and the three changes that genuinely move the number.

Fluency and correctness are different dials

Hallucination is the name for model output that is fluent, confident, correctly formatted, and false. The word is a poor metaphor, because it suggests a malfunction. Nothing malfunctioned. A system that predicts plausible continuations produced a plausible continuation, and there is no separate step where it checks whether that continuation is true.

Picture two dials.

Does this read like an expert wrote it?

The entire training objective.

Turned up as far as it goes, on every kind of text, billions of times.

Do the claims match reality?

Never directly measured.

Where true statements were common in the training text, the plausible continuation happens to be the correct one. Where they were rare, it is not — and the first dial has not moved an inch.

Training turns the first one all the way up. The second moves only as a side effect.

That is what catches people out. With a person, hesitation is a signal — a guesser slows down, hedges, looks away — and it is informative because it is costly. For a model, hedging is a writing style: a confident sentence and a cautious one cost exactly the same to produce.

Why the invention is so well shaped

Random noise would be easy to catch. What you get instead is correctly shaped, because shape is what the model learned best. A Python keyword argument is lowercase words joined by underscores. A citation is author, year, title, journal, volume, pages. A DOI starts with 10. and a slash. Those patterns are everywhere in the training text, so they are known cold — far better known than any rare fact that happens to wear one of them. When the fact is missing, the shape is still there, and the shape is what gets filled in.

Think of a forger who has studied ten thousand passports and never one from the country you asked about. What he hands you has a photo page, a machine-readable strip and a serial number in the right format. It will be far more convincing than a blank page, and wrong in the one way that matters.

It also explains why the invented thing is so often the thing that ought to exist. A library with to_json, from_json and to_csv is a pattern with an obvious gap in it, so when you ask how to write YAML, to_yaml is exactly what the pattern predicts — and exactly what you get. The model is not reporting on the library. It is completing a series.

Where it bites hardest

A fact is roughly as safe as it was common in the training text. The capital of France survives any amount of paraphrase; a rare, specific string does not. These are the categories where that bites:

Citations and references — papers, sections, page numbers, DOIs, ISBNs. Every field is a format the model knows and a value it does not.
Identifiers — ticket numbers, order ids, commit hashes, vulnerability ids, package names, URLs. Unique by design, which is another way of saying unlearnable.
API surfaces — function names, keyword arguments, CLI flags, config keys, especially anything that changed between versions.
Numbers — prices, dates, counts, percentages, benchmarks. A number is generated the same way a word is.
Anything private — your codebase, your schema, your customer's contract, last quarter's figures. Never in the training text at all, so there is nothing to recall and only shape to fall back on.
Anything recent — events after the training data ends.

A rule of thumb: the more exactly a thing has to match to be worth anything, the less you should accept it unchecked. A paraphrase of a well-known idea has slack, so small errors stay harmless. A URL, a key name or a version number has none — one character off and it is wrong.

Why "be accurate" changes almost nothing

Your instruction arrives in the same channel as everything else and is treated the same way: as text that conditions what comes next. It can shift tone, register and format. What it cannot do is hand the model something to check against.

"Do not make things up" asks for a filter on a property the model has no access to. From the inside there is no difference between a fact seen ten thousand times and a phrase assembled a moment ago — both are high-probability continuations, and neither carries a flag saying which it is.

Worse, a firm accuracy instruction can make the failure harder to catch: push hard on confidence and you suppress the hedging language, the one weak signal you had. You get the same wrong answers with the qualifiers stripped off. Asking "are you sure?" fails for the same reason — that is another plausible continuation, and agreement is the plausible continuation of a challenge, so a model will often reverse an answer that was right.

Bad — demands accuracy the model cannot establish, and gets a fluent answer either way.

Good — puts the answer inside the request, so the job is reading rather than recall.

The first prompt contains no documentation at all, so "be accurate" is an instruction with nothing behind it. What it costs you is a wrong key name delivered in the tone of someone who looked it up.

Three changes actually move the number, and they only work together.

Put the facts in the request
Turns a closed-book exam into an open-book one, opened at the right page. Recall is hard and unverifiable; finding a sentence in a document you were just handed is neither.
Make refusing an allowed answer
Otherwise, when the answer is not in the source, the model bridges the gap with something that fits the surrounding text beautifully.
Make the claims checkable by code
A verbatim quote and a source id per claim turns verification into a string comparison — not another model call, and not a careful human read.

Each one is load-bearing for the next. Grounding without a refusal route makes things worse.

Fix one: put the facts in the request

Grounding means supplying the source text in the request and requiring the answer to come from it. It turns the closed-book exam into an open-book one — and not merely open-book, but open at the right page. Recall is hard and unverifiable; finding a sentence in a document you were just handed is much easier and far likelier to be right.

Two details decide whether this works or half-works. First, the instruction has to be exclusive. "Using only the source below" is a different instruction from "using the source below". Without the word only, the model blends what you gave it with what it recalls, and the blend is the worst outcome available: mostly correct, quietly wrong in one place, wearing the authority of your document.

Second, the answer has to actually be in the source. If it is not, and refusing is not possible, you have made things worse — the model will bridge the gap with something that fits the surrounding text beautifully. Grounding without a refusal route is a trap, which is why the next fix is not optional.

Fix two: make refusing an allowed answer

Refusing is genuinely hard for a model, and the reason matters. In nearly all the text it learned from, a question is followed by an answer; "I don't know" is rare, and where it appears it often reads like a failure. So every prompt carries an instruction you did not write: produce something. Unless you say otherwise, the highest-probability continuation of a question is an answer, whether or not the material for one exists.

The fix is to make not-answering a first-class output: name the exact string, state its condition, and forbid the half-measure.

Bad — asks for a judgement the model cannot make, returned as prose your code cannot read.

Good — names one exact output and the condition for it.

"If you are not sure" gets you "I believe the value is around 30 seconds, though you may want to confirm" — a hallucination wearing a hedge, which your parser treats as an answer and your reader believes.

Fix three: make claims checkable by code

The third fix changes what you ask for so that a machine can test the answer: prefer output whose truth can be decided by ordinary code against something you already have. Once you are grounded you have the source, so the check is a string comparison — not another model call, not a careful human read. The form that does it is a verbatim quote and a source id on every claim.

Bad — produces authoritative citations a human must check one at a time, against the world.

Good — asks for the same claims plus the evidence, in fields you can test.

Now the verification is ordinary software, and it is short:

"Section 3.2" may not exist, and finding out costs a person a minute per claim — so nobody does it after the third one. The quote version answers the same question in microseconds and cannot be talked round.

Be clear about what this proves. A matching quote shows the sentence is real and really in that report. It does not show the claim follows from it — a real sentence can still be summarised into something it does not say. What it removes is the whole class of invented sources, where most of the damage is.

What the three fixes do not do

They lower the rate. They do not take it to zero, and no prompt does. A grounded, refusal-enabled, quote-checked answer can still misread a real source, quote accurately and frame the quote wrongly, or refuse a question the document did answer. They change the odds; that is all.

So the last move is not a prompt at all — it is designing the feature so that being wrong is survivable. Show the source beside the answer, so a reader checks in two seconds instead of trusting. Keep the action reversible. Put a person in front of anything expensive, public or irreversible. Where a wrong answer costs a shrug, lean on the model; where it costs a customer, a human stays in the loop.

And you cannot tell whether any of it helped by reading a few outputs and feeling better. Knowing your rate went from one in five to one in fifty needs a fixed set of inputs and a score — the subject of Measuring Instead of Vibing later on.

A cheat sheet to keep

Where to go next

You have the mechanism and what follows from it: fluency is free and correctness is not, invention arrives correctly formatted, rare and exact strings are where it lands, and "be accurate" has nothing behind it. Against that, three moves that work — ground the answer, allow a refusal, demand evidence a machine can check.

All three lean on one assumption: that the facts are in the request. Giving the Model Your Data takes that apart. It answers what this lesson did not — who puts the facts there, and how you choose between pasting them in, fetching them at query time, or handing the model a tool to fetch them.

Before that, run the experiment. Take a document you have — a README, a policy, an incident report — and ask a question about it cold, with no source attached. Note the answer. Then paste the document in, add "using only the source below" and a refusal string, and ask something the document genuinely does not cover. If you get the refusal, your instruction is exclusive enough. If you get a confident answer instead, you have just watched the gap being bridged — and you will recognise it everywhere now.