An Eval Harness You Can Trust
Building an evaluation you would bet a release on: sample size, variance across runs, statistical significance, and why a two-point score improvement is usually noise.
Building an evaluation you would bet a release on: sample size, variance across runs, statistical significance, and why a two-point score improvement is usually noise.
The pull request has an eval report attached, which is more than most of them manage. Eighty-four percent on the baseline, eighty-six on the branch, fifty cases, one run each. Two points up, nothing red, ship it.
A week later you re-run the baseline — same prompt, same fifty cases, nothing touched — and it comes back at eighty-eight percent. The improvement you shipped was smaller than the noise you never measured. This lesson is the statistics that would have caught that before the merge: what a single run actually tells you, how wide a pass rate really is, how many cases a difference of a given size needs before it is visible at all, and the one change to how you compare two versions that buys you more resolution than any amount of extra data.
The number your harness printed is not a property of your prompt. It is one draw from a distribution — the spread of results the same suite would produce if you ran it over and over, and how likely each one is. You saw one value from that spread and wrote it in a pull request as though it were a constant.
Three things move it, and they behave differently.
Model variance is the same prompt and the same input producing different output on two calls; the foundations course covered why, under sampling and temperature. Set variance is that your fifty cases are one possible fifty — a different fifty, curated by the same rules from the same failures, would score differently because it happens to contain harder or easier work. Both of these shrink as the number of cases grows, and shrink in a way you can calculate.
The third does not shrink at all. Run-level variance is everything that changed between two runs that was not your prompt: a retrieval index that reindexed overnight, a provider serving a slightly different build behind the same interface, a grading model whose own version moved. It hits every case in a run together, so adding cases does not average it away. This is why a baseline number from three weeks ago is not a baseline. It is a historical fact about a system that no longer exists.
Here is the same suite, same prompt, five runs, nothing changed between them:
run passed rate
1 42/50 84%
2 44/50 88%
3 41/50 82%
4 44/50 88%
5 43/50 86%Six points from bottom to top, from a system standing perfectly still. The two-point improvement in the pull request sits comfortably inside the range this harness produces while doing nothing.
A pass rate is a proportion — a count of successes over a count of trials — and proportions come with a width you can compute rather than guess at.
The standard error is the typical distance between the rate
you measured and the rate you would get from unlimited cases.
For a proportion it is the square root of p * (1 - p) / n,
where p is the observed rate and n the number of cases. At
85% on 50 cases that is 5.0 percentage points. One standard
error. The typical miss, not the worst one.
Two standard errors either side gives you roughly a 95% confidence interval: a range computed from your data such that, if you repeated the whole procedure many times, 95% of the intervals you produced would contain the true rate. (That is the honest phrasing. It is a statement about the procedure, not a 95% chance that the truth is inside this particular range.)
Do not compute it as p ± 2 * SE. That form, the Wald
interval, misbehaves exactly where eval work lives: it runs
past 100% on a strong result, and at 50 out of 50 passes it
reports a width of zero, which is plainly false. Use the
Wilson interval, which is the same idea with the arithmetic
done properly:
import math
def wilson(passes: int, n: int, z: float = 1.96) -> tuple:
"""95% confidence interval for a pass rate."""
if n == 0:
return (0.0, 1.0)
rate = passes / n
denom = 1 + z * z / n
centre = (rate + z * z / (2 * n)) / denom
spread = z / denom * math.sqrt(
rate * (1 - rate) / n + z * z / (4 * n * n)
)
return (centre - spread, centre + spread)Run it across sizes at a realistic 85% and the shape of the problem appears:
cases rate 95% interval half-width
20 85% 62% - 95% ±16 points
50 85% 72% - 93% ±10 points
100 85% 77% - 91% ±7 points
200 85% 79% - 89% ±5 points
500 85% 82% - 88% ±3 points
1000 85% 83% - 87% ±2 pointsFifty cases cannot resolve anything finer than about ten points. That is the whole reason the opening scene happened.
Bad — prints a delta as a fact, so it gets approved as one.
delta = new_rate - old_rate
print(f"pass rate {old_rate:.0%} -> {new_rate:.0%} "
f"({delta:+.0%})")Good — prints each rate with the range it is actually consistent with.
old_low, old_high = wilson(old_passes, n)
new_low, new_high = wilson(new_passes, n)
print(f"baseline {old_rate:.0%} 95% CI "
f"[{old_low:.0%}, {old_high:.0%}]")
print(f"branch {new_rate:.0%} 95% CI "
f"[{new_low:.0%}, {new_high:.0%}]")The first version says 84% -> 86% (+2%) and a reviewer reads
an improvement. The second says 84% [71%, 92%] against
86% [74%, 93%], and the same run now plainly says the two
versions are indistinguishable — which is true, and which nobody
merges on.
Turn the question around. Instead of asking what your suite scored, ask what size of difference it could see at all. That is the minimum detectable effect, and it is a property of the suite, fixed before you run anything.
Three terms first. The null hypothesis is the assumption you are trying to rule out: that the two versions are identical and the gap you see is noise. A p-value is the probability of a gap at least this large if the null hypothesis were true — a small p means noise is a poor explanation, and the convention is that below 0.05 you stop offering it. Statistical power is the probability your test finds a real difference of a given size when one exists; 80% is the usual target, and it means one real improvement in five goes unnoticed.
Those three plus your baseline rate fix the number of cases:
def cases_needed(base: float, lift: float) -> int:
"""Cases per arm, 95% significance, 80% power."""
z_sum = 1.96 + 0.8416 # significance + power
new = base + lift
variance = base * (1 - base) + new * (1 - new)
return math.ceil(z_sum ** 2 * variance / lift ** 2)From a baseline of 85%, comparing two independent runs:
to detect cases per arm total runs
+2 points ~4,700 ~9,400
+5 points ~680 ~1,360
+10 points ~140 ~280Nearly five thousand cases per side to see two points. Nobody has that, nobody will build it, and this is the calculation that makes teams conclude evaluation is hopeless below enormous scale.
It is not hopeless. That table describes an unpaired comparison — two independent groups, compared as two averages. It is the wrong test, and you have been running it by accident.
Look again at where the variance in that calculation comes from. Most of it is case difficulty: some of your cases are hard and some are easy, and the mix dominates the score. Now notice that when you run both versions on the same golden set, that mix is identical on both sides. You are paying, statistically, for uncertainty you do not have.
A paired comparison runs both versions over the same inputs and compares case by case rather than total against total. Everything shared between the arms cancels exactly. What survives is only where they disagree.
Lay the two runs out as a table of four counts:
new passes new fails
old passes a b
old fails c da and d are cases both versions get the same way. They carry
no information about which version is better — a case both
prompts pass tells you nothing about the difference between the
prompts. Only b and c do, and a case the two versions
disagree on is a discordant pair.
McNemar's test is the test built on exactly that: it ignores
a and d, and asks whether the split between b and c is
lopsided enough to be more than a coin. In eval work the
discordant counts are small, so use the exact version — a
binomial sign test over b + c trials, which asks how often
pure chance would put this many of the disagreements on one
side:
from math import comb
def mcnemar_exact(broke: int, fixed: int) -> float:
"""One-sided p-value: is the new version better?
broke = old passed and new failed.
fixed = old failed and new passed.
"""
discordant = broke + fixed
if discordant == 0:
return 1.0
tail = sum(comb(discordant, k)
for k in range(fixed, discordant + 1))
return tail / 2 ** discordantNow the part worth remembering. Here are two branches. Both score 42/50 on the baseline and 47/50 on the branch. Both report a headline of 84% to 94%, up ten points. The headline cannot tell them apart. The paired table can:
branch A branch B
pass fail pass fail
pass 42 0 pass 39 3
fail 5 3 fail 8 0
fixed 5, broke 0 fixed 8, broke 3
p = 0.031 p = 0.113Branch A moved five cases and broke nothing: chance produces a clean five-nil sweep once in thirty-two tries, so noise is a bad explanation and the change is real. Branch B fixed more cases and broke three, and eight-to-three is the kind of split a coin delivers about one run in nine. Same fifty cases, same headline, one genuine improvement and one shrug — and the only thing that separates them is that you kept the per-case results.
For scale: on that same data the unpaired two-proportion test gives p = 0.11 for both branches. Pairing turned a non-result into a verdict without adding a single case.
Pairing has preconditions, and they are all about the harness. Identical inputs, identical pinned corpus version, identical grader version, and the two arms run interleaved in the same session so that any provider-side drift lands on both equally. And above all, per-case results that survive the run.
Bad — stores a summary, so no later comparison can ever be paired.
results = [run_case(case) for case in golden_set]
db.insert("eval_runs", {
"run_id": run_id,
"prompt_version": prompt_version,
"passed": sum(results),
"total": len(results),
})Good — stores one row per case per run, so any two runs join on case id.
db.insert_many("eval_case_results", [
{"run_id": run_id,
"prompt_version": prompt_version,
"case_id": case.id,
"klass": case.klass,
"attempt": attempt,
"passed": passed}
for case, attempt, passed in results
])The summary row discards precisely the column the paired test runs on, and the loss is not recoverable: to compare against last month you have to re-run last month's prompt at full token cost, assuming you still have it. Two integers per case, kept from the start, are the difference between a comparison and an archaeology project.
A discordant pair is still one observation. Branch A's five fixed cases might be five real fixes, or four fixes and one case that was always a coin landing differently. To tell those apart, run each case more than once.
With k attempts, a case stops being a boolean and becomes a
rate. Three cases from a five-attempt run:
case attempts verdict
refund-after-cancel 5 / 5 passes
escalate-legal-threat 0 / 5 fails
ambiguous-plan-upgrade 2 / 5 unstableThe third is not a fail and not a flaky test to be re-run until it behaves. It is a case whose behaviour your prompt does not actually determine, and reporting it as either outcome throws that away. Instability is a finding: track the count of unstable cases as its own number, because it moves when the system gets less predictable, which is often before the pass rate moves at all.
Repeats cost k times the tokens, so do not pay for them
everywhere. Cases at 5/5 and 0/5 tell you nothing new on the
sixth attempt. Run every case once, take the discordant set, and
spend your repeats there — cost is roughly n + k * (b + c)
rather than k * n. Re-run branch A that way and the five fixed
cases resolve: four go 5/5 for the new prompt, one goes 3/5, and
that last one was never fixed.
Bad — gives a second attempt only to the failures, so noise can only ever push the score up.
results = {case.id: run_case(case) for case in golden_set}
for case_id, passed in list(results.items()):
if not passed:
results[case_id] = run_case(by_id(case_id))Good — every case gets the same number of attempts, and the attempts are kept.
results = {
case.id: [run_case(case) for _ in range(ATTEMPTS)]
for case in golden_set
}A case that genuinely passes half the time scores as a pass three-quarters of the time under that retry, while a case that never passes stays honest — so the inflation lands entirely on the unstable cases, which are the ones you most needed to see. Worse, the score climbs a little each time someone adds another retry, and that reads exactly like progress.
The last piece is a control. An A/A test runs the baseline against itself as if it were a second arm: same prompt, same cases, interleaved with the real comparison. Its true delta is zero by construction, so whatever it reports is pure noise, at today's provider, on today's index, with today's grader.
You already report per failure class rather than one average. What changes at this level is what the statistics do to those slices, and it cuts both ways.
First, the intervals get very wide. A class with twelve cases that goes from 12/12 to 10/12 looks alarming and is not measurable: the Wilson intervals are 76%–100% and 55%–95%. They overlap across almost their entire length. That slice cannot distinguish a real regression from two coins.
Second, you are now running one comparison per class, and every comparison gets its own chance of a false alarm. This is the multiple comparisons problem. With eight classes tested at the conventional 5%, the chance that at least one of them fires on a run where nothing changed is one minus 0.95 to the eighth — about 34%. A third of your clean runs will show a regression somewhere.
The resolution is to decide in advance what the run is for. Nominate a primary comparison before you run: the class the change was aimed at. That one gets the paired test and the verdict. Every other class is a screen — it says "look here", and looking means a follow-up run with repeats on that class, not a blocked merge.
The must-never classes are outside this entirely. Zero failures on irreversible actions is not a hypothesis you are testing, it is a property you are asserting, and one failure is one too many whatever an interval says. No statistics wanted there, and none needed.
Then the opposite failure, the one a headline hides:
class n base new delta discordant
stale-policy 12 12/12 9/12 -3 0 fixed, 3 broke
out-of-scope 10 7/10 10/10 +3 3 fixed, 0 broke
(five others) 48 43/48 43/48 0 unchanged
overall 70 62/70 62/70 0The overall line did not move by a single case. Underneath it, one class collapsed and another improved by the same amount. Each move alone is p = 0.125 — a coin gives you three-nil once in eight tries — so neither is a result on its own. But two opposite three-case swings, in the two classes the change touched, on a run where nothing else moved, is not something to average away. It is the thing to spend your repeats on, and the overall number would have told you the branch was inert.
WHAT A NUMBER IS
one run a sample from a distribution
model + set variance shrink as n grows
run-level variance does not shrink; re-run the baseline
stale baseline a fact about a system that is gone
WIDTH OF A PASS RATE
standard error sqrt(p * (1 - p) / n)
85% on 50 cases SE 5 points, 95% CI about ±10
use Wilson Wald breaks at 0%, 100%, and small n
half-width at 85% n=50 ±10 n=200 ±5 n=1000 ±2
never print a delta without the interval beside it
UNPAIRED SAMPLE SIZE (85% base, 95% sig, 80% power)
+2 points ~4,700 cases per arm
+5 points ~680 cases per arm
+10 points ~140 cases per arm
read as stop comparing two independent runs
PAIRED, ON THE SAME INPUTS
b = old passed, new failed c = old failed, new passed
agreements (a, d) carry no information; ignore them
McNemar exact sign test over b + c trials
5-0 sweep p = 0.031, a result on 50 cases
8-3 split p = 0.113, same headline, no result
real sample size b + c, not n
need ~25 discordant at 4-in-5 win rate
~50 discordant at 7-in-10
requires same inputs, pinned corpus, same
grader, interleaved runs, per-case rows
REPEATS
k attempts per case a case becomes a rate, not a boolean
5/5 and 0/5 settled; do not spend more on them
2/5 unstable — a finding, not a re-run
spend adaptively n + k * (b + c), not k * n
never retry only the failures; it only ever
inflates, and only the unstable cases
A/A arm baseline vs itself, interleaved; its
delta is your noise floor
SLICING
class of 12 10/12 vs 12/12 is not measurable
8 classes at 5% ~34% chance of a false alarm per run
primary comparison named before the run; gets the verdict
every other class a screen: triggers a look, not a block
must-never classes asserted, not tested; zero is zero
flat overall can hide equal moves in opposite
directions; check the discordant columnYou can now say what a run means. One run is a sample; a pass rate carries an interval you print beside it; a two-point move on fifty cases is noise and the arithmetic says so before the argument starts. You compare on the same inputs and read the disagreements, which turns a suite too small to prove anything into one that can. You spend repeats where the disagreements are, you keep an A/A arm to know your floor, and you name the comparison that counts before the numbers arrive.
All of that assumed the grader is a fixed instrument. It is usually a model, and then it is a third source of variance with biases of its own — one that favours the longer answer, or the answer in the first position, or its own output. LLM-as-Judge and Its Failure Modes, next in this course, is about building a judge that tracks human labels and noticing the day it stops. Note that a judge whose version changes between your two arms breaks pairing outright, because the thing you held constant was not.
Go and do the cheapest experiment here. Run your suite five times against a prompt you have not touched, write down the five numbers, and take the difference between the largest and the smallest. Every improvement you have shipped that was smaller than that gap was a coin, and you now know which ones they were.