What a Language Model Actually Does
Next-token prediction, and how that one mechanism explains almost every behaviour you will meet: fluency without understanding, confidence when wrong, and why the same question can get two different answers.
Next-token prediction, and how that one mechanism explains almost every behaviour you will meet: fluency without understanding, confidence when wrong, and why the same question can get two different answers.
You ask a model for help with a date-formatting bug and it hands
back six lines of clean, commented code that call
datetime.format_iso(). The name is perfect. The arguments are
exactly the ones you would have picked. The function does not
exist.
Nothing malfunctioned. That answer is the machine working precisely as designed, and by the end of this lesson you will be able to predict it — along with most of the other behaviour that surprises people. There is one mechanism underneath all of it, it is simpler than you expect, and once you can see it working you stop being caught out by it.
A language model does exactly one thing. It takes the text that exists so far, and produces a score for every possible next piece of text — how likely each one is to come next. Something picks one of them, sticks it on the end, and the whole process runs again on the slightly longer text.
That is called next-token prediction. A token is a piece of text roughly the size of a short word, or a fragment of a longer one; models read and write in tokens rather than letters. How text gets cut into them, and why the count ends up on your bill, is the subject of the next lesson. For now, read "token" as "the next little piece of text".
Here is one step of the loop, with made-up but realistic numbers:
Text so far: "The capital of France is"
Scores for the next token:
" Paris" 0.87
" located" 0.04
" the" 0.03
" a" 0.02
" Lyon" 0.001
...and a score for every other token it knowsSomething picks " Paris", and the loop runs again on the
slightly longer text.
Read everything so far
The whole text, every time. Nothing is remembered between passes — the text is the memory.
Score every possible next token
Not a shortlist. A number for every token it knows, produced in one pass through billions of parameters.
Pick one and add it to the end
Which one gets picked is a separate decision, and the subject of a whole lesson later.
Picture someone writing on a long strip of paper. They can read everything already on the strip, they must add one more piece, and they are never allowed to go back and change what is behind them. That is the whole job.
If it only guesses the next piece of text, why is the output ever useful? Because of how the guessing was learned.
Training works like this. Take an enormous amount of written text. Hide the next piece, ask the model to guess it, compare the guess with what was actually there, and nudge the model's parameters — the billions of numbers that turn input text into scores — a tiny amount in the direction that would have made the real answer more likely. Then do that again, an unimaginable number of times.
Nobody typed in the fact that Paris is the capital of France. Guessing well across billions of passages is impossible unless those numbers settle into an arrangement that behaves as though it knows a great deal about how the world gets described. Facts, grammar, code idioms, the shape of an argument, the layout of a recipe — all of it arrives as a side effect of getting good at one guessing game.
Think of a footpath worn across a field. Nobody surveyed it and nobody wrote it down. It is the shape left behind by a very large number of people walking mostly the same way.
One more stage matters, because otherwise the model would be strange to talk to. A pure text-continuer, handed your question, would often continue it with three more questions — that is what follows a question in a lot of real documents. So after the main training, models get a second and much smaller round: examples of helpful replies written by people, plus human judgements about which of two candidate answers is better. That is what turns a text-continuer into something that answers you.
It changes what the model finds likely. It does not change the mechanism. It is still scoring tokens and picking one.
Nothing in that loop checks whether anything is true. The target it was trained against was "make the piece of text that really came next more likely" — never "be right".
Truth gets in sideways. Most of the training text was written by people trying to be accurate, so accurate continuations are usually the likely ones. That correlation is the entire reason the tool is useful. The places where it breaks down are where the tool fails.
Go back to datetime.format_iso(). In a great deal of real
text, a sentence like "you can format the timestamp with" is
followed by a module name, a dot, a verb-shaped function name,
and a pair of parentheses. The model produced the most plausible
thing of that shape.
A property of the text.
Does this read like something a knowledgeable person would have written? That is exactly what the training target measured, billions of times.
A property of the world.
Nothing in the loop checks it. Truth gets in sideways — most training text was written by people trying to be accurate, so accurate continuations are usually the likely ones.
Where that correlation breaks down is where the tool fails.
So why does wrong output sound so sure of itself?
Because certainty and hedging are token patterns like any other. Explanations in the training text are mostly written in a confident register — documentation, textbooks, answers from people who knew. A confident register is therefore the likely continuation of an explanation, whatever the specific content happens to be. There is no separate channel where the model tells you how sure it is. The tone comes out of the same loop as the facts, and it is generated with no reference to them.
Two things follow. The first is that you cannot read confidence as evidence. The second catches almost everybody.
Bad — checks the answer by asking the thing that produced it.
You: Which rsync flag deletes files at the destination that
no longer exist at the source?
AI: Use --purge-missing.
You: Are you sure?
AI: Apologies, you are right to question that. The correct
flag is --remove-orphans.Good — checks the answer against something that is not the model.
rsync --help | grep -i delete # ask the tool itself
man rsync # read the real description"Are you sure?" is just more text, and the likely continuation
of being challenged is an apology and a revision — so you get
one whether the first answer was wrong or right. The bad version
talks you out of correct answers exactly as readily as wrong
ones, and either way you finish with two answers and no evidence
for either. Neither of those flags exists, by the way; the real
one is --delete, which one run of the good version tells you
in under a second.
Since every token is chosen as a continuation of everything before it, the words you send are not a query being looked up in an index. They are the opening of the document the model is completing. Every word in them moves the scores.
That makes phrasing a control, not a courtesy. Think of your words as coordinates: they set down where in the vast space of possible text the continuation begins. "Explain how caching works" starts in a neighbourhood full of general introductions. "Explain how caching works to a team debugging stale prices on a checkout page" starts somewhere else entirely, and the text that plausibly follows is different text.
Bad — leaves the model to guess what kind of document this is.
Is this query slow?
SELECT * FROM orders WHERE customer_email = 'ana@example.com';Good — names the situation, so the plausible continuation is the useful one.
Postgres, orders table, about 4 million rows, indexed on id
only. This runs on every page load and takes 900ms. Explain
what is making it slow.What most plausibly follows a generic question is a generic
answer, so the first version earns you a tidy paragraph about
the usual reasons queries are slow — and it will not mention the
missing index on customer_email, because nothing in the input
said the index was missing. You lose the answer to a fact you
already had in your head and did not type. Building prompts
deliberately is the whole of the lesson "Writing a Prompt That
Holds Up"; what matters here is the mechanism that makes
phrasing matter at all.
Every token the model produces becomes part of the input for the next one. The output is fed back in as it is written, and that one detail explains a lot.
An early wrong turn does not get corrected — it gets built on. If the twelfth token commits to a library that does not exist, the most plausible continuation is more text about that library: a convincing install command, convincing arguments, a convincing note about a common pitfall. The model is not inventing a cover-up. Each step really is the most reasonable continuation of a document whose opening happened to be wrong.
The same mechanism explains the opposite effect, which you can use. When you ask for the reasoning worked through rather than the conclusion alone, the intermediate steps become part of the text the later tokens are predicted from. A conclusion that follows five lines of correct working is a different prediction problem from a conclusion that follows nothing at all.
It also explains why the same question can come back two different ways. Reword it and you have moved the starting point, so the scores change. Send byte-identical input twice and you can still get different answers, because the step that picks a token among the high-scoring candidates normally involves deliberate randomness — and one different pick early sends the rest of the document somewhere else. That knob, and when to turn it down, is the lesson "Sampling, Temperature, and Non-Determinism".
Everything in this lesson comes from one loop. Here is the loop, and everything that falls out of it:
THE MECHANISM
one loop score every possible next token, pick
one, append it, run the whole thing again
no plan nothing is decided before the first
token is chosen
training hide the next piece, guess, nudge the
parameters, repeat enormously
tuning a later, smaller stage makes it answer
you instead of continuing you
WHAT FOLLOWS FROM IT
fluent != correct plausible shape is what got optimised;
truth only came along for the ride
confidence is tone produced by the same loop as the content,
never a report of how sure it is
"are you sure?" a challenge, and the likely continuation
of a challenge is a retraction - either way
phrasing steers your words open the document, they are
not a lookup key
errors compound a wrong early token gets conditioned on,
not corrected
answers vary different wording, and randomness in the
pick, both move the result
WHAT TO DO ABOUT IT
verify outside docs, a compiler, a test, the tool's own
--help - never the model that said it
state the situation domain, versions, constraints, audience,
what you already ruled out
ask for the working reasoning in the text improves what
follows the reasoning
re-ask, don't argue a better opening beats three rounds of
correctionYou now have the one fact the rest of this course rests on: a model scores the next token, picks one, appends it, and repeats. Fluency without understanding, confidence without certainty, and sensitivity to a single reworded sentence are not quirks bolted onto that mechanism. They are that mechanism, seen from different angles.
Next is Tokens, Context, and Why They Cost You. This lesson kept saying "the text so far" as though it were free and endless. It is neither: that lesson tells you what a token really is, how much text a model can hold at once, what happens to a conversation when it runs out of room, and why the same chat gets more expensive the longer it goes on.
Before you move on, run one experiment. Pick a question in an area you know well — well enough to grade the answer yourself. Ask it once in a single vague sentence, then ask it again with the situation spelled out: the versions, the constraints, what you have already ruled out. Read the two answers side by side. Then take whichever one is correct and reply "Are you sure?", and watch a right answer get apologised away. Ten minutes of that will teach you more about the mechanism than another ten pages of description.