Sampling, Temperature, and Non-Determinism
Why a model that is deterministic underneath gives you different answers anyway: temperature, top-p, seeds, and how to decide when variation is a feature and when it is a bug you have to design around.
Why a model that is deterministic underneath gives you different answers anyway: temperature, top-p, seeds, and how to decide when variation is a feature and when it is a bug you have to design around.
You run your review classifier over two hundred real support messages and every label is right. The next morning you run the same two hundred through the same code, and three come back with different labels. You changed nothing — not the prompt, not the input file, not a line of code — and the temperature is already zero.
Nothing is broken. You have met the layer between the model and the text you see: something has to choose a word, and the way it chooses is a set of dials you own. By the end of this lesson you will know what those dials do, why turning them all the way down buys you no guarantee, and how to tell a feature that should vary from one that must not.
You already know the model works by predicting what comes next. What is easy to miss is that it never predicts a next piece of text — it predicts all of them at once, with a number attached to each.
Every run produces one raw score — the usual name is a logit — for every entry in the model's vocabulary. Those scores are squashed into a probability distribution: every possible next token with a probability beside it, all adding up to 1. (A token is the small chunk of text a model works in; "Tokens, Context, and Why They Cost You" teaches that properly.)
Suppose the text so far is Tomorrow the weather will be. The
model's answer is not "sunny". It is closer to this:
That distribution is the model's entire output: a menu, not a decision. Something downstream still has to pick one item off it, append it, and run the model again for the next one. That picker is the decoding strategy, and almost every surprising thing about model output — the repetition, the creativity, the flakiness — happens there rather than inside the model. Given identical input and identical arithmetic, the model produces the same distribution every time; everything that varies, varies after this point.
The simplest picker takes the highest number on the list, every single time. That is greedy decoding: same input, same distribution, same top entry, same output.
So why is it not the default everywhere? Because always taking the favourite is a trap in long text. The model's most likely continuation of a sentence it has just written is often a variation of that sentence, so greedy decoding walks into loops:
The service was fine. The service was fine. The service was
fine. The service was fine.Each repeat really is the highest-probability choice at that instant, and nothing breaks the cycle, because nothing ever varies. The picker that takes no risks writes the dullest possible text and sometimes jams.
The alternative is sampling: treat the numbers as odds and roll a weighted die.
Always take the highest number.
Same input, same distribution, same output — every time.
And the model's most likely continuation of a sentence it has just written is often that sentence again, so long text walks into loops with nothing to break them.
Roll a weighted die.
Roughly 47 runs in 100 give "sunny", 35 "cloudy", 16 "raining", 3 "snowing".
The favourite still usually wins — it no longer always wins.
Temperature is the dial that decides how lopsided the die is. Mechanically it divides every raw score by the temperature before the numbers become probabilities. Small number, bigger differences; large number, smaller differences.
Easier to see than to say — same four candidates, four temperatures:
token T=0.2 T=0.7 T=1.0 T=1.5
sunny 0.815 0.533 0.469 0.408
cloudy 0.182 0.347 0.347 0.334
raining 0.003 0.111 0.156 0.196
snowing 0.000 0.010 0.028 0.063Read the rows, not only the columns. Temperature never changes
which token is in front — "sunny" leads at every setting. It
changes how much the leader wins by. At T=0.2 the favourite
takes four rolls in five and "snowing" has vanished; at T=1.5
the field is nearly level and the outsider turns up once every
sixteen rolls.
Two things follow. First, temperature is not a creativity dial. It cannot make the model think of something better; it only shifts weight toward options it already ranked lower. Near the top that means fresher phrasing; further down, nonsense.
Second, temperature 0 is not a separate mode. Dividing by zero is undefined, so a request for temperature 0 is handled as "make the leader's advantage overwhelming", which comes out the same as greedy decoding. Reach for it whenever you want the same answer twice — with the caveat two sections below.
Temperature has a side effect. Because it reshapes the whole distribution, raising it also lifts the long tail — thousands of tokens each near-impossible but not collectively so. Any one of them derails the sentence.
Top-p sampling (also called nucleus sampling) fixes that
by shortening the menu before the die is rolled. Sort the
candidates from most to least likely, add up probabilities from
the top, and stop as soon as the running total reaches p.
Everything past that point is discarded and the roll happens
among the survivors.
At T=1.0, with top_p set to 0.9:
sunny 0.469 cumulative 0.469 kept
cloudy 0.347 cumulative 0.816 kept
raining 0.156 cumulative 0.972 kept, crosses 0.9
snowing 0.028 — discarded
everything else discardedThe clever part is that the surviving group breathes with the model's confidence. When the model is sure — the closing bracket of a JSON object, the second half of a stock phrase — one candidate already holds 0.9 alone, so top-p leaves a menu of one and you are back to greedy decoding. When the model is genuinely torn, the group is wide and you get variety exactly where it is harmless.
You will also meet top-k, the older idea: keep a fixed number of candidates — say 40 — whatever their probabilities. It is blunter, because 40 is too many when the model is certain and too few when it is not.
Back to the three labels that flipped overnight, temperature at zero, every input identical. No die was rolled, so where did the difference come from? From arithmetic that is very slightly not repeatable. Three things stack up.
Adding numbers in a different order differs
Decimals have limited precision, so (a + b) + c and
a + (b + c) can differ in the last few digits. One
distribution means adding millions of such values across
thousands of parallel units.
How the work is split depends on the machine
And your request shares that machine with other people's — providers batch requests together, so the shape of the computation depends on who else is served in the same millisecond.
Near-ties turn a last digit into a whole word
Usually the wobble changes nothing, because the leader leads by a mile. But two tokens at 0.4999 and 0.5001 have their winner decided by it.
And one different word changes everything after
Every prediction after it happens in a different context, so one coin flip early on produces a whole different paragraph.
A fourth cause has nothing to do with hardware: the model behind an endpoint can be updated, so your output moves on a day you shipped nothing.
A seed is a starting number for the pseudo-random generator that rolls the die. Give the same seed twice and you get the same sequence of rolls, which means the same choices from the same distributions.
That is useful, and narrower than it sounds: a seed fixes the die and nothing else. Every cause in the previous section is still in play, because those happen before the roll — they change the numbers the die is rolled against. A seed makes two runs comparable; it does not make them identical. Use it where comparability is the point: changing one line of a prompt and wanting the effect of that line rather than of luck, or reproducing a bad output to debug it.
Now the decision this lesson exists for, and it is not a technical one. Ask who reads the output.
If a human reads it and chooses, variation is doing work for you. Five subject lines to pick between beat the same one five times, and a brainstorm that returns an identical list every run is a broken brainstorm. Drafting, naming, rewriting — in all of them you would notice the absence of variation as a defect. Run warm on purpose.
The mistake here is not picking the wrong number. It is picking one number for the whole application.
Bad — one temperature in config, shared by every call the app makes.
# config.py
TEMPERATURE = 0.9
def summarise(review_text):
return generate(SUMMARY_PROMPT + review_text,
temperature=TEMPERATURE)
def classify(review_text):
return generate(CLASSIFY_PROMPT + review_text,
temperature=TEMPERATURE)Good — each call sets the temperature its own job needs.
def summarise(review_text):
# a person reads this and picks; spread is welcome
return generate(SUMMARY_PROMPT + review_text,
temperature=0.8)
def classify(review_text):
# a switch statement reads this; spread is a defect
return generate(CLASSIFY_PROMPT + review_text,
temperature=0.0)One shared number is always wrong for one of the two callers.
Turn it down and every summary arrives in the same flat
phrasing; leave it at 0.9 and one review in fifty comes back
labelled Positive! instead of positive, your switch falls
through to the default branch, and the ticket routes nowhere.
Temperature belongs to the task, not to the application.
If a program reads the output, variation is a fault waiting for a busy Tuesday. Classification, extraction, routing, tool selection, anything that becomes a database row or a branch in your code — one answer is correct and every other roll of the die is wrong by definition.
Turn the temperature down for those, but do not stop there: low temperature is a strong preference, not a lock. Design so that occasional variation is survivable.
Constrain the output shape rather than hoping for it. Asking for one of a fixed set of labels, or a schema you can validate, turns "the model said something unexpected" from a silent wrong answer into a check that fails loudly — "Structured Output Instead of Prose" is that move.
Then normalise before you compare: strip whitespace, lowercase, map to your enum. Half the variation that breaks real systems is a trailing full stop or a capital letter, and absorbing it costs one line.
Tests are where this bites first, because the usual instinct — record what came out, assert it comes out again — is exactly what the model will not promise you.
Bad — pins the exact sentence the model happened to produce the day the test was written.
def test_summary_reports_the_double_charge():
summary = summarise(
"Charged twice and support never replied."
)
assert summary == (
"The customer was charged twice and did not hear "
"back from support."
)Good — asserts the property the feature actually promises.
def test_summary_reports_the_double_charge():
summary = summarise(
"Charged twice and support never replied."
)
assert "twice" in summary.lower()
assert len(summary) < 200The first test fails on a run where nothing changed, so somebody
re-records the expected string to get the build green, then does
it again, then deletes the test. The cost is not the red build:
it is that on the day summarise starts dropping the complaint
altogether, the test meant to catch that is gone. Assert what
must be true of every acceptable answer, and let the wording
move. One bad result proves little either way: it might be the
one roll in fifty.
THE PIPELINE
model -> distribution over all tokens -> picker -> one token
every dial below lives in the picker, not inside the model
THE DIALS (they belong to the call, not to the application)
temperature 0 take the favourite; repeatable, can loop
temperature ~0.2 parsed output, classification, extraction
temperature ~0.7 prose a person reads
temperature >1.2 mostly incoherence
top_p 1.0 neutral: no candidates cut
top_p 0.9 drop the tail, keep the plausible group
top_k 40 older, blunter: fixed candidate count
seed same die rolls; NOT same output
tune ONE of temperature or top_p, leave the other neutral
WHO READS THE OUTPUT?
a human picking -> variation is the product, run warm
a program parsing -> variation is a bug, run cold + validate
STILL VARIES AT TEMPERATURE 0, BECAUSE
float addition order changes with hardware scheduling
batching mixes your request with other people's
near-ties amplify a last-digit wobble into a new paragraph
the model behind the endpoint can be updated under you
SO
normalise before comparing (trim, lowercase, map to enum)
assert properties in tests, never exact strings
store output you will need again; do not re-request it
never key a cache or an identity check on response textThe next question is the one this lesson deliberately left open: how much does your output vary, and is a change an improvement or noise? That is "Measuring Instead of Vibing" — a fixed set of real inputs, an expected outcome for each, and a number you can compare between runs. This lesson says variation exists; that one hands you the instrument to see it.
Before you move on, go and feel it. Take one prompt whose answer you can judge at a glance, run it five times at temperature 0, five at 1.0 and five at 1.5, and read the fifteen results side by side. The point where the answers stop being different and start being wrong is the number you have been guessing at.