Testing Non-Deterministic Systems
When the same input gives a different answer, equality assertions stop working. Scoring instead of matching, thresholds over sample sets, and detecting drift.
When the same input gives a different answer, equality assertions stop working. Scoring instead of matching, thresholds over sample sets, and detecting drift.
Every technique so far rested on one assumption: the same input produces the same output, so a test can compare against an expected value. Language models, recommendation engines, ranking systems, OCR and speech recognition all break that assumption by design.
assert output == expected stops working, and the usual reactions are both
wrong: give up and test nothing, or pin an exact output and watch the test
fail on every model update. By the end of this lesson you will know the
third option — scoring instead of matching, thresholds over sample sets, and
detecting drift.
Several different things get called non-determinism, and they need different treatment.
Remove the variation.
An unseeded random number generator — seed it. A clock — freeze it. Set or dictionary ordering — sort before comparing.
A race in your own code is not non-determinism at all. It is a bug, and the answer is to fix it.
The variation is the product working.
A language model, even at temperature 0. Ranking over a changing corpus. OCR, speech recognition and image classification, which produce confidence scores rather than certainties.
And anything with a third-party model behind it that the provider can update without telling you.
The shift is from asking "is the output equal to this?" to "does the output have the properties it must have?"
# fails on every model update, and often between runs
def test_summary():
result = summarise(ARTICLE)
assert result == "The council approved the budget on Tuesday."
# properties that must hold for any acceptable answer
def test_a_summary_is_short_grounded_and_on_topic():
result = summarise(ARTICLE)
assert 20 <= len(result.split()) <= 60 # length
assert "budget" in result.lower() # key term
assert not contains_url(result) # format
assert every_number_in(result) in numbers_of(ARTICLE) # groundedThat last assertion is the most valuable pattern in this lesson. Every number in the summary must appear in the source — a mechanical, deterministic check on a non-deterministic output that catches invented figures, which is one of the most damaging failure modes there is.
The general shape: find the properties that any correct answer shares, and assert on those. Most of them turn out to be checkable exactly.
structural valid JSON; the required fields present and typed;
within a length range; no forbidden content
grounded every number, name, date and quoted phrase appears
in the provided source
behavioural a refusal for a request that should be refused;
a tool called with the right arguments
semantic close enough in meaning to a reference answerOnly the last needs anything fancy, and the first three catch most real regressions.
When meaning is what matters, compare meaning rather than characters.
def test_the_answer_means_the_same_as_the_reference():
result = answer("When was the budget approved?")
reference = "The budget was approved on Tuesday."
assert cosine_similarity(embed(result), embed(reference)) > 0.85Embedding similarity is cheap, deterministic given a fixed embedding model, and tolerant of rewording. It is also crude: two sentences with opposite meanings can score highly if they share vocabulary, so it is a smoke test rather than a judgement.
The stronger tool is an LLM judge — a model asked to evaluate an output against criteria:
JUDGE_PROMPT = """
Question: {question}
Reference answer: {reference}
Candidate answer: {candidate}
Does the candidate contain the same factual claims as the
reference? Answer with JSON: {{"same_facts": true|false,
"missing": [...], "invented": [...]}}
"""Judges are useful and have their own failure modes: they prefer longer answers, they favour their own style, they are inconsistent near a boundary, and they can be led by the phrasing of the criteria. Treat a judge as a noisy instrument — calibrate it against human labels on a sample before trusting it, and keep the deterministic checks as the primary gate.
One sample of a random process tells you very little. The unit of testing becomes a set, with a threshold.
Bad — one sample, one assertion, a flaky test:
def test_the_classifier_is_correct():
assert classify("this is terrible") == "negative"Good — a labelled set and a threshold:
CASES = load_labelled_cases("tests/data/sentiment.jsonl") # 200 cases
def test_the_classifier_meets_its_accuracy_threshold():
results = [(classify(case.text), case.label) for case in CASES]
accuracy = sum(a == b for a, b in results) / len(results)
assert accuracy >= 0.92, f"accuracy {accuracy:.3f} below 0.92"
def test_no_regression_on_the_cases_that_previously_passed():
"""The known-good set must stay at 100%."""
for case in load_labelled_cases("tests/data/must_pass.jsonl"):
assert classify(case.text) == case.labelThe bad test asserts on one call to a probabilistic system. It passes most of the time and fails occasionally, so it enters the retry-and-move-on cycle from the flakiness lesson — and once it does, it can never report a real regression, because a real regression looks exactly like its usual noise.
The good version measures accuracy over two hundred cases, which is stable enough to assert on, and keeps a separate must-pass set: cases that matter individually and are expected to be right every time. That split is the practical answer — a statistical threshold for the population, and exact assertions for the small set of cases you cannot get wrong.
Two details make thresholds work. Set the threshold below current performance with a margin — at 0.94 today, gate at 0.92 — so ordinary noise does not fail the build. And report the number every run, so a slow decline from 0.94 to 0.925 is visible before it crosses.
The failure mode that has no equivalent in deterministic testing: nothing in your code changes and the behaviour does, because something outside it moved.
model drift the provider updated the model behind an unversioned
name, or deprecated the version you pinned
data drift the input distribution changed — new slang, a new
product category, a different customer segment
concept drift the right answer changed. "Spam" means something
different than it did two years ago.
prompt drift an edit intended for one case degrades othersThe defences:
[ ] pin the model version explicitly, never a floating alias
[ ] run the evaluation set on a schedule, not only on code change
— this is the only way drift is caught, since nothing triggers
a build
[ ] track the score over time as a graph, not a pass or fail
[ ] monitor the INPUT distribution as well as the output
[ ] alert on a change in refusal rate, average output length, or
tool-call frequency — cheap proxies that move when a model does
[ ] keep the evaluation set fresh: add production failures to itThe second is the important one. A drifted model breaks nothing in your repository, so no pull request triggers a check. A nightly or weekly evaluation run is the only thing that notices.
Even where the core is non-deterministic, most of the system is not, and testing the deterministic parts deterministically is where the cheap coverage is.
prompt construction given this input and context, the prompt
built is exactly this. A pure function.
parsing the response given this response text, the parsed object
is this. Deterministic, and where a lot of
real bugs live.
retrieval given this query and corpus, these documents
are returned in this order
tool dispatch given this tool call, this function runs with
these arguments
error handling a malformed response, a refusal, a timeout,
a truncated output — all deterministicThen use recorded responses for everything above the model call, exactly as the faking lesson described. A recorded set of real model outputs, replayed, gives you a fast, deterministic suite for the whole system around the model — and one small, slow, scheduled suite that actually calls it.
@pytest.mark.parametrize("recording", load_recordings("summarise"))
def test_summary_parsing_and_validation(recording):
"""Deterministic: replayed real outputs, real parsing code."""
result = parse_summary_response(recording.raw_response)
assert result.text
assert result.token_count == recording.expected_tokensThat split — deterministic tests on recordings for the pipeline, a scheduled evaluation for the model — is the arrangement that works. It keeps the fast gate fast, and puts the statistical work where its cost is affordable.
# Separate the two kinds
CONTROLLABLE -> remove it
unseeded RNG -> seed / clock -> freeze / set ordering -> sort /
a race in your code -> a BUG, fix it
INHERENT -> test differently
language models (even at temperature 0), ranking over a changing
corpus, OCR, speech, classification confidence, any third-party
model that can be updated
# Temperature 0 is NOT determinism
# batching, hardware and floating-point non-associativity still
# change the output. Providers say so.
# Score, do not match
structural valid JSON, required fields, length range, no
forbidden content
GROUNDED every number, name, date and quotation appears in the
source <- the most valuable check; mechanical, exact,
and catches invented figures
behavioural refuses what it should refuse; calls the right tool
with the right arguments
semantic close enough in meaning to a reference
# the first three are exact checks on a fuzzy output
# Semantic comparison
embedding similarity cheap, deterministic, crude — a smoke test.
Opposite meanings can score high.
an LLM judge stronger, and a NOISY INSTRUMENT: prefers
long answers, favours its own style,
inconsistent near boundaries. Calibrate
against human labels; keep deterministic
checks as the primary gate.
# Test the DISTRIBUTION, not the run
one sample of a probabilistic system is a flaky test — and once it
is treated as flaky it can never report a real regression
a labelled set (100-1000 cases) + a THRESHOLD
a separate MUST-PASS set, expected right every time
set the threshold BELOW current performance with a margin
report the number every run, so a slow decline is visible
# Drift — nothing in your code changed and the behaviour did
model drift the provider updated it, or deprecated your version
data drift the input distribution changed
concept drift the right answer changed
prompt drift an edit for one case degraded others
defences
pin the model version, never a floating alias
run the eval set ON A SCHEDULE — no pull request will trigger it
track the score as a GRAPH, not a pass/fail
monitor the input distribution too
alert on refusal rate, output length, tool-call frequency
add production failures to the eval set
# Version everything that affects the output
model + version, prompt, corpus, temperature, eval set — all in
version control, all recorded with each result. Otherwise a score
change cannot be attributed.
# Most of the system IS deterministic — test it that way
prompt construction / response parsing / retrieval order / tool
dispatch / error handling: malformed, refusal, timeout, truncation
# use RECORDED responses for the pipeline: a fast deterministic
# suite around the model, plus one small scheduled suite that
# really calls itThe last lesson of this track is about people rather than systems: leading quality across a team, where a specialist adds most, and coaching rather than gatekeeping.
Before that, take one non-deterministic behaviour you work with and write down three properties any correct output must have. At least one of them will turn out to be exactly checkable, and that assertion is worth more than any number of attempts to pin the output itself.