beginner25 minby DevFox

Sampling, Temperature, and Non-Determinism

Why a model that is deterministic underneath gives you different answers anyway: temperature, top-p, seeds, and how to decide when variation is a feature and when it is a bug you have to design around.

You run your review classifier over two hundred real support messages and every label is right. The next morning you run the same two hundred through the same code, and three come back with different labels. You changed nothing — not the prompt, not the input file, not a line of code — and the temperature is already zero.

Nothing is broken. You have met the layer between the model and the text you see: something has to choose a word, and the way it chooses is a set of dials you own. By the end of this lesson you will know what those dials do, why turning them all the way down buys you no guarantee, and how to tell a feature that should vary from one that must not.

The model hands you a distribution, not a word

You already know the model works by predicting what comes next. What is easy to miss is that it never predicts a next piece of text — it predicts all of them at once, with a number attached to each.

Every run produces one raw score — the usual name is a logit — for every entry in the model's vocabulary. Those scores are squashed into a probability distribution: every possible next token with a probability beside it, all adding up to 1. (A token is the small chunk of text a model works in; "Tokens, Context, and Why They Cost You" teaches that properly.)

Suppose the text so far is Tomorrow the weather will be. The model's answer is not "sunny". It is closer to this:

sunny0%

cloudy0%

raining0%

snowing0%

Plus a very long tail of every other token it knows, each with a number of its own

That distribution is the model's entire output: a menu, not a decision. Something downstream still has to pick one item off it, append it, and run the model again for the next one. That picker is the decoding strategy, and almost every surprising thing about model output — the repetition, the creativity, the flakiness — happens there rather than inside the model. Given identical input and identical arithmetic, the model produces the same distribution every time; everything that varies, varies after this point.

Greedy: always take the favourite

The simplest picker takes the highest number on the list, every single time. That is greedy decoding: same input, same distribution, same top entry, same output.

So why is it not the default everywhere? Because always taking the favourite is a trap in long text. The model's most likely continuation of a sentence it has just written is often a variation of that sentence, so greedy decoding walks into loops:

Each repeat really is the highest-probability choice at that instant, and nothing breaks the cycle, because nothing ever varies. The picker that takes no risks writes the dullest possible text and sometimes jams.

The alternative is sampling: treat the numbers as odds and roll a weighted die.

Greedy

Always take the highest number.

Same input, same distribution, same output — every time.

And the model's most likely continuation of a sentence it has just written is often that sentence again, so long text walks into loops with nothing to break them.

Sampling

Roll a weighted die.

Roughly 47 runs in 100 give "sunny", 35 "cloudy", 16 "raining", 3 "snowing".

The favourite still usually wins — it no longer always wins.

Same model, same distribution, different picker. Everything else in this lesson follows from this choice.

Temperature: how much the favourite wins by

Temperature is the dial that decides how lopsided the die is. Mechanically it divides every raw score by the temperature before the numbers become probabilities. Small number, bigger differences; large number, smaller differences.

Easier to see than to say — same four candidates, four temperatures:

Read the rows, not only the columns. Temperature never changes which token is in front — "sunny" leads at every setting. It changes how much the leader wins by. At T=0.2 the favourite takes four rolls in five and "snowing" has vanished; at T=1.5 the field is nearly level and the outsider turns up once every sixteen rolls.

Two things follow. First, temperature is not a creativity dial. It cannot make the model think of something better; it only shifts weight toward options it already ranked lower. Near the top that means fresher phrasing; further down, nonsense.

Second, temperature 0 is not a separate mode. Dividing by zero is undefined, so a request for temperature 0 is handled as "make the leader's advantage overwhelming", which comes out the same as greedy decoding. Reach for it whenever you want the same answer twice — with the caveat two sections below.

Top-p: cutting the tail before you roll

Temperature has a side effect. Because it reshapes the whole distribution, raising it also lifts the long tail — thousands of tokens each near-impossible but not collectively so. Any one of them derails the sentence.

Top-p sampling (also called nucleus sampling) fixes that by shortening the menu before the die is rolled. Sort the candidates from most to least likely, add up probabilities from the top, and stop as soon as the running total reaches p. Everything past that point is discarded and the roll happens among the survivors.

At T=1.0, with top_p set to 0.9:

The clever part is that the surviving group breathes with the model's confidence. When the model is sure — the closing bracket of a JSON object, the second half of a stock phrase — one candidate already holds 0.9 alone, so top-p leaves a menu of one and you are back to greedy decoding. When the model is genuinely torn, the group is wide and you get variety exactly where it is harmless.

You will also meet top-k, the older idea: keep a fixed number of candidates — say 40 — whatever their probabilities. It is blunter, because 40 is too many when the model is certain and too few when it is not.

Why temperature zero still is not a promise

Back to the three labels that flipped overnight, temperature at zero, every input identical. No die was rolled, so where did the difference come from? From arithmetic that is very slightly not repeatable. Three things stack up.

Adding numbers in a different order differs
Decimals have limited precision, so (a + b) + c and a + (b + c) can differ in the last few digits. One distribution means adding millions of such values across thousands of parallel units.
How the work is split depends on the machine
And your request shares that machine with other people's — providers batch requests together, so the shape of the computation depends on who else is served in the same millisecond.
Near-ties turn a last digit into a whole word
Usually the wobble changes nothing, because the leader leads by a mile. But two tokens at 0.4999 and 0.5001 have their winner decided by it.
And one different word changes everything after
Every prediction after it happens in a different context, so one coin flip early on produces a whole different paragraph.

This is why your flips landed on the ambiguous reviews: borderline for you is borderline inside the model too

A fourth cause has nothing to do with hardware: the model behind an endpoint can be updated, so your output moves on a day you shipped nothing.

What a seed actually fixes

A seed is a starting number for the pseudo-random generator that rolls the die. Give the same seed twice and you get the same sequence of rolls, which means the same choices from the same distributions.

That is useful, and narrower than it sounds: a seed fixes the die and nothing else. Every cause in the previous section is still in play, because those happen before the roll — they change the numbers the die is rolled against. A seed makes two runs comparable; it does not make them identical. Use it where comparability is the point: changing one line of a prompt and wanting the effect of that line rather than of luck, or reproducing a bad output to debug it.

When variation is the product

Now the decision this lesson exists for, and it is not a technical one. Ask who reads the output.

If a human reads it and chooses, variation is doing work for you. Five subject lines to pick between beat the same one five times, and a brainstorm that returns an identical list every run is a broken brainstorm. Drafting, naming, rewriting — in all of them you would notice the absence of variation as a defect. Run warm on purpose.

The mistake here is not picking the wrong number. It is picking one number for the whole application.

Bad — one temperature in config, shared by every call the app makes.

Good — each call sets the temperature its own job needs.

One shared number is always wrong for one of the two callers. Turn it down and every summary arrives in the same flat phrasing; leave it at 0.9 and one review in fifty comes back labelled Positive! instead of positive, your switch falls through to the default branch, and the ticket routes nowhere. Temperature belongs to the task, not to the application.

When it is a bug you design around

If a program reads the output, variation is a fault waiting for a busy Tuesday. Classification, extraction, routing, tool selection, anything that becomes a database row or a branch in your code — one answer is correct and every other roll of the die is wrong by definition.

Turn the temperature down for those, but do not stop there: low temperature is a strong preference, not a lock. Design so that occasional variation is survivable.

Constrain the output shape rather than hoping for it. Asking for one of a fixed set of labels, or a schema you can validate, turns "the model said something unexpected" from a silent wrong answer into a check that fails loudly — "Structured Output Instead of Prose" is that move.

Then normalise before you compare: strip whitespace, lowercase, map to your enum. Half the variation that breaks real systems is a trailing full stop or a capital letter, and absorbing it costs one line.

Testing something that will not sit still

Tests are where this bites first, because the usual instinct — record what came out, assert it comes out again — is exactly what the model will not promise you.

Bad — pins the exact sentence the model happened to produce the day the test was written.

Good — asserts the property the feature actually promises.

The first test fails on a run where nothing changed, so somebody re-records the expected string to get the build green, then does it again, then deletes the test. The cost is not the red build: it is that on the day summarise starts dropping the complaint altogether, the test meant to catch that is gone. Assert what must be true of every acceptable answer, and let the wording move. One bad result proves little either way: it might be the one roll in fifty.

A cheat sheet to keep

Where to go next

The next question is the one this lesson deliberately left open: how much does your output vary, and is a change an improvement or noise? That is "Measuring Instead of Vibing" — a fixed set of real inputs, an expected outcome for each, and a number you can compare between runs. This lesson says variation exists; that one hands you the instrument to see it.

Before you move on, go and feel it. Take one prompt whose answer you can judge at a glance, run it five times at temperature 0, five at 1.0 and five at 1.5, and read the fifteen results side by side. The point where the answers stop being different and start being wrong is the number you have been guessing at.