beginner25 minby DevFox

What a Language Model Actually Does

Next-token prediction, and how that one mechanism explains almost every behaviour you will meet: fluency without understanding, confidence when wrong, and why the same question can get two different answers.

You ask a model for help with a date-formatting bug and it hands back six lines of clean, commented code that call datetime.format_iso(). The name is perfect. The arguments are exactly the ones you would have picked. The function does not exist.

Nothing malfunctioned. That answer is the machine working precisely as designed, and by the end of this lesson you will be able to predict it — along with most of the other behaviour that surprises people. There is one mechanism underneath all of it, it is simpler than you expect, and once you can see it working you stop being caught out by it.

One mechanism, all the way down

A language model does exactly one thing. It takes the text that exists so far, and produces a score for every possible next piece of text — how likely each one is to come next. Something picks one of them, sticks it on the end, and the whole process runs again on the slightly longer text.

That is called next-token prediction. A token is a piece of text roughly the size of a short word, or a fragment of a longer one; models read and write in tokens rather than letters. How text gets cut into them, and why the count ends up on your bill, is the subject of the next lesson. For now, read "token" as "the next little piece of text".

Here is one step of the loop, with made-up but realistic numbers:

text

Text so far:  "The capital of France is"

Scores for the next token:
  " Paris"          0.87
  " located"        0.04
  " the"            0.03
  " a"              0.02
  " Lyon"           0.001
  ...and a score for every other token it knows

Something picks " Paris", and the loop runs again on the slightly longer text.

Read everything so far
The whole text, every time. Nothing is remembered between passes — the text is the memory.
Score every possible next token
Not a shortlist. A number for every token it knows, produced in one pass through billions of parameters.
Pick one and add it to the end
Which one gets picked is a separate decision, and the subject of a whole lesson later.

An answer three hundred words long is this loop run several hundred times

Picture someone writing on a long strip of paper. They can read everything already on the strip, they must add one more piece, and they are never allowed to go back and change what is behind them. That is the whole job.

Where the scores come from

If it only guesses the next piece of text, why is the output ever useful? Because of how the guessing was learned.

Training works like this. Take an enormous amount of written text. Hide the next piece, ask the model to guess it, compare the guess with what was actually there, and nudge the model's parameters — the billions of numbers that turn input text into scores — a tiny amount in the direction that would have made the real answer more likely. Then do that again, an unimaginable number of times.

Nobody typed in the fact that Paris is the capital of France. Guessing well across billions of passages is impossible unless those numbers settle into an arrangement that behaves as though it knows a great deal about how the world gets described. Facts, grammar, code idioms, the shape of an argument, the layout of a recipe — all of it arrives as a side effect of getting good at one guessing game.

Think of a footpath worn across a field. Nobody surveyed it and nobody wrote it down. It is the shape left behind by a very large number of people walking mostly the same way.

One more stage matters, because otherwise the model would be strange to talk to. A pure text-continuer, handed your question, would often continue it with three more questions — that is what follows a question in a lot of real documents. So after the main training, models get a second and much smaller round: examples of helpful replies written by people, plus human judgements about which of two candidate answers is better. That is what turns a text-continuer into something that answers you.

It changes what the model finds likely. It does not change the mechanism. It is still scoring tokens and picking one.

Fluent and correct are different properties

Nothing in that loop checks whether anything is true. The target it was trained against was "make the piece of text that really came next more likely" — never "be right".

Truth gets in sideways. Most of the training text was written by people trying to be accurate, so accurate continuations are usually the likely ones. That correlation is the entire reason the tool is useful. The places where it breaks down are where the tool fails.

Go back to datetime.format_iso(). In a great deal of real text, a sentence like "you can format the timestamp with" is followed by a module name, a dot, a verb-shaped function name, and a pair of parentheses. The model produced the most plausible thing of that shape.

Fluency

A property of the text.

Does this read like something a knowledgeable person would have written? That is exactly what the training target measured, billions of times.

Truth

A property of the world.

Nothing in the loop checks it. Truth gets in sideways — most training text was written by people trying to be accurate, so accurate continuations are usually the likely ones.

Where that correlation breaks down is where the tool fails.

Only one of these was ever being optimised, and it is not the one you care about

Confidence is a writing style

So why does wrong output sound so sure of itself?

Because certainty and hedging are token patterns like any other. Explanations in the training text are mostly written in a confident register — documentation, textbooks, answers from people who knew. A confident register is therefore the likely continuation of an explanation, whatever the specific content happens to be. There is no separate channel where the model tells you how sure it is. The tone comes out of the same loop as the facts, and it is generated with no reference to them.

Two things follow. The first is that you cannot read confidence as evidence. The second catches almost everybody.

Bad — checks the answer by asking the thing that produced it.

Good — checks the answer against something that is not the model.

"Are you sure?" is just more text, and the likely continuation of being challenged is an apology and a revision — so you get one whether the first answer was wrong or right. The bad version talks you out of correct answers exactly as readily as wrong ones, and either way you finish with two answers and no evidence for either. Neither of those flags exists, by the way; the real one is --delete, which one run of the good version tells you in under a second.

Your prompt is the start of the text

Since every token is chosen as a continuation of everything before it, the words you send are not a query being looked up in an index. They are the opening of the document the model is completing. Every word in them moves the scores.

That makes phrasing a control, not a courtesy. Think of your words as coordinates: they set down where in the vast space of possible text the continuation begins. "Explain how caching works" starts in a neighbourhood full of general introductions. "Explain how caching works to a team debugging stale prices on a checkout page" starts somewhere else entirely, and the text that plausibly follows is different text.

Bad — leaves the model to guess what kind of document this is.

Good — names the situation, so the plausible continuation is the useful one.

What most plausibly follows a generic question is a generic answer, so the first version earns you a tidy paragraph about the usual reasons queries are slow — and it will not mention the missing index on customer_email, because nothing in the input said the index was missing. You lose the answer to a fact you already had in your head and did not type. Building prompts deliberately is the whole of the lesson "Writing a Prompt That Holds Up"; what matters here is the mechanism that makes phrasing matter at all.

It writes forward and cannot take it back

Every token the model produces becomes part of the input for the next one. The output is fed back in as it is written, and that one detail explains a lot.

An early wrong turn does not get corrected — it gets built on. If the twelfth token commits to a library that does not exist, the most plausible continuation is more text about that library: a convincing install command, convincing arguments, a convincing note about a common pitfall. The model is not inventing a cover-up. Each step really is the most reasonable continuation of a document whose opening happened to be wrong.

The same mechanism explains the opposite effect, which you can use. When you ask for the reasoning worked through rather than the conclusion alone, the intermediate steps become part of the text the later tokens are predicted from. A conclusion that follows five lines of correct working is a different prediction problem from a conclusion that follows nothing at all.

It also explains why the same question can come back two different ways. Reword it and you have moved the starting point, so the scores change. Send byte-identical input twice and you can still get different answers, because the step that picks a token among the high-scoring candidates normally involves deliberate randomness — and one different pick early sends the rest of the document somewhere else. That knob, and when to turn it down, is the lesson "Sampling, Temperature, and Non-Determinism".

A cheat sheet to keep

Everything in this lesson comes from one loop. Here is the loop, and everything that falls out of it:

Where to go next

You now have the one fact the rest of this course rests on: a model scores the next token, picks one, appends it, and repeats. Fluency without understanding, confidence without certainty, and sensitivity to a single reworded sentence are not quirks bolted onto that mechanism. They are that mechanism, seen from different angles.

Next is Tokens, Context, and Why They Cost You. This lesson kept saying "the text so far" as though it were free and endless. It is neither: that lesson tells you what a token really is, how much text a model can hold at once, what happens to a conversation when it runs out of room, and why the same chat gets more expensive the longer it goes on.

Before you move on, run one experiment. Pick a question in an area you know well — well enough to grade the answer yourself. Ask it once in a single vague sentence, then ask it again with the situation spelled out: the versions, the constraints, what you have already ruled out. Read the two answers side by side. Then take whichever one is correct and reply "Are you sure?", and watch a right answer get apologised away. Ten minutes of that will teach you more about the mechanism than another ten pages of description.