Evaluating an AI Feature
Golden sets, rubrics, and regression suites that run in CI. Choosing metrics that move when quality moves, and keeping an evaluation honest as the feature it measures changes.
Golden sets, rubrics, and regression suites that run in CI. Choosing metrics that move when quality moves, and keeping an evaluation honest as the feature it measures changes.
Someone tightens the system prompt on a Friday afternoon. The twenty cases in your spreadsheet still pass — nineteen of them, the same nineteen as last week — so it ships. On Monday, support has four tickets about the assistant confidently quoting a refund window that was retired two years ago.
The spreadsheet was right about the method and wrong about the scale. Twenty cases and a pass rate got you off vibes, which is the whole point of "Measuring Instead of Vibing" in the beginner course. This lesson turns that afternoon's work into something a team runs on every change: a curated golden set that grows from real failures, rubrics that do not drift, metrics that move when quality moves, per-stage evaluation that tells you what broke, and a CI setup that blocks the merges worth blocking.
A spreadsheet has one reader and one moment. An eval suite has to survive being run by someone who did not write it, six months later, against a feature that changed underneath it. Three properties get you there, and none are about scoring. It lives in the repository next to the code it measures, so a change to expected behaviour shows up in a diff and gets reviewed. It runs unattended, which means every input it needs is pinned rather than assumed. And each case carries its provenance, so the next person can tell whether it guards something real or was somebody's idea of a nice example in 2024.
That makes a case a file, not a row. Here is one from a help-desk assistant that answers billing questions from a help centre and returns structured output:
# evals/cases/refund-after-cancellation.yaml
id: refund-after-cancellation
category: stale-policy # the failure class it guards
added: 2026-03-14
provenance: ticket 88213 — assistant quoted a retired policy
input:
question: "I cancelled Pro yesterday. Can I still get a refund?"
account:
plan: pro
cancelled_days_ago: 1
corpus_version: help-centre@2026-03-10 # pinned, so it repeats
checks:
- kind: cites
article: refunds-and-cancellations
- kind: must_not_contain
text: "30-day" # the retired window
- kind: rubric
id: answer-qualityThe category and provenance lines are the ones people skip
and the ones that make the set maintainable: a case with no
class cannot be reported on, and a case with no history cannot
be retired with confidence.
A golden set is the fixed collection of inputs your feature is evaluated against, each with an expected outcome. The instinct when growing one is to reach for production traffic and take a big sample. That instinct produces a set that looks impressive and tests almost nothing.
Bad — samples traffic, so the set inherits traffic's shape.
recent = load_production_requests(days=30)
cases = random.sample(recent, 200)Good — a quota per failure class, filled on purpose.
QUOTAS = {
"stale-policy": 12, # answers from retired documentation
"must-escalate": 15, # money, legal, or an angry customer
"ambiguous-plan": 10, # question that spans two plans
"out-of-scope": 10, # should refuse, not improvise
"hostile-input": 8, # instructions hidden in the question
"happy-path": 15, # the boring majority, still checked
}
cases = [case
for klass, limit in QUOTAS.items()
for case in pick_reviewed(recent, klass=klass, n=limit)]Real traffic is overwhelmingly easy. The cases that cost you money are rare by definition, so in a random two hundred they show up once or twice; a regression that breaks all of them moves the overall pass rate by half a point, and nobody looks twice at half a point.
The reframe that makes curation obvious: your suite is a specification of behaviour, not a survey of users. You are not estimating how often the feature is right in the wild — that is what production monitoring is for — you are asserting that specific things remain true. So over-represent everything you are afraid of: irreversible actions, the boundary between answering and refusing, inputs that are ambiguous even to a human, inputs actively trying to break you.
Curated does not mean invented. The best cases are ones reality handed you, and the discipline is the same one you already apply to bugs in ordinary code: a failure that reached users gets a regression case before it gets a fix.
Three rules keep that from degenerating. Freeze the input exactly as it arrived, including account state and corpus version — a paraphrase is a different question, and models are sensitive to the difference. Record what actually happened alongside what should have happened; the wrong answer is evidence, and later it explains why the case exists. And assign it a class. If it fits none of them, you have found a new failure class, which is worth more than the case itself.
The set grows by failures and shrinks by retirement — never by someone adding a few more examples on a quiet afternoon. Those cases have no class, no provenance, and nothing to say when they fail.
Plenty of checks are exact. Did the response validate against the schema, does the cited article id exist, was the escalation flag set. Those need no judgement and should never be graded by a human or a model.
What is left is the part where judgement is unavoidable: is this answer actually useful to the person who asked. A rubric is how you make that judgement repeatable, and the shape of the rubric decides whether it is.
Bad — one number for the whole answer, so nothing is ever comparable twice.
RUBRIC = """
Rate the overall quality of this answer from 1 to 5,
where 5 is excellent and 1 is poor.
"""
PASS_THRESHOLD = 4Good — three independent questions, each answerable yes or no from the output alone.
RUBRIC = [
"Does the answer state an outcome — eligible, not "
"eligible, or escalated — rather than summarising the "
"refund policy in general?",
"Is every factual claim traceable to one of the cited "
"articles?",
"Does it name the condition the outcome depends on, such "
"as the plan or the cancellation date?",
]
# the case passes only if all three are yesCompresses, then drifts.
Several independent judgements collapse into one number, so an answer that gained a citation and lost the outcome scores the same as one that did the reverse. The trade is invisible.
And the scale moves: a weak 4 to one grader is a strong 3 to the next, and your own 4 in March is a 3 in June.
Stays comparable.
Each answerable from the output alone, and each failure names which property was missing.
Binary questions drift far less, because there is nowhere to drift to.
Three rules follow. Keep it to three to five questions — past that, graders skim. Make each answerable from the output alone, without holding an ideal answer in mind, since "is it as good as it could be" is not a question two people answer the same way. And version the rubric: the moment you reword a question, scores from before and after stop being comparable, so bump a number and say so rather than quietly breaking your own history.
Someone has to apply it. Early on that is a person, and a few dozen cases with three questions each is genuinely an hour's work. Handing the reading to a model is the obvious scaling move and brings its own distortions — the advanced course covers those in "LLM-as-Judge and Its Failure Modes". Either way, sanity-check the rubric on humans first: give the same ten outputs to two people and compare. If they disagree on a question, the question is wrong, not the people.
Because your set is curated, its absolute pass rate means very little — a set full of hard cases scores worse than a set full of easy ones, and neither number describes your users' experience. What matters is movement, and movement per class matters more than movement overall.
Bad — one average, so any improvement pays for any failure.
pass_rate = passed / total
assert pass_rate >= 0.85Good — a floor on the classes that must never break, and no class allowed to slide.
assert failures(klass="must-escalate") == 0
for klass in QUOTAS:
assert rate(klass) >= baseline[klass] - 0.02The average is the problem. Fifteen easy cases improving buys exactly enough headroom to hide the one case where the assistant told a customer they were owed money they were not, and the build stays green while the worst thing your feature can do starts happening again.
Reporting per class is what makes a regression legible:
category n pass baseline delta
must-escalate 15 15/15 15/15 0
stale-policy 12 10/12 12/12 -2 <-- regression
ambiguous-plan 10 9/10 9/10 0
out-of-scope 10 10/10 8/10 +2
hostile-input 8 8/8 8/8 0
happy-path 15 14/15 15/15 -1
overall 70 66/70 67/70 -1The overall line moved by one case. Underneath it, the change made refusals better and stale-policy answers worse — a real trade someone has to decide about, which the summary number would have shrugged off as noise.
Now the harder version: the metric that improves while users complain. A metric is a proxy, and optimising a proxy is not optimising the thing. You loosen the escalation threshold, so auto-resolution climbs and the escalation rate falls — both look like wins until you notice the escalations you stopped sending were the ones that needed sending. You add "be concise" and the conciseness question starts passing everywhere, because the model dropped the caveat that made the answer correct.
The defence is a counter-metric: every metric that can be gamed by doing more of something gets paired with one that punishes doing it badly. Coverage pairs with correctness. Escalation rate pairs with escalation precision — of the ones you escalated, how many needed it. Conciseness pairs with the question about naming the condition.
An end-to-end score tells you something broke. It does not tell you what, and on a feature with retrieval, generation and a response contract, "what" has at least three answers that need completely different fixes.
So evaluate the stages separately, with the boundary between them pinned. Component evaluation means each stage is measured against fixed inputs, so a failure can only be caused by that stage:
def evaluate_stages(case):
hits = retrieve(case.question, corpus=case.corpus_version)
retrieved = [hit.article_id for hit in hits[:5]]
# generation is judged against a stored context, so a bad
# answer here can never be retrieval's fault
answer = generate(case.question, context=case.fixed_context)
return {
"retrieval": set(case.expected_articles) <= set(retrieved),
"contract": validates(answer, ResponseSchema)
and all(article_exists(a) for a in
answer.citations),
"generation": grade_with_rubric(answer, RUBRIC),
}Read the three together and the diagnosis is immediate. Retrieval red, generation green: the model answered well from what it was given, and the articles it needed never arrived — fixing that is the subject of "Retrieval That Actually Retrieves" earlier in this course. Retrieval green, generation red: the material was there and the answer still went wrong, so the problem is the prompt or the model. Contract red alone: nothing is wrong with the thinking, your parser is about to throw, and that is the cheapest failure here to fix.
Keep a small end-to-end suite as well. Stage tests use pinned contexts, so they are blind to the bugs that only appear once the real stages are wired together — a retriever returning three articles where the prompt expects one, a formatter truncating a citation list. Ten end-to-end cases across your riskiest classes catch that, and they are the ones you run last.
The suite that catches everything is too slow and expensive to run on every push; the suite that runs on every push catches less than you want. Split it in two, because gating and reporting are genuinely different jobs.
The gate is fast, deterministic and small. Exact checks only — schema validity, citation ids, the escalation flag, the must-never classes. No rubric grading, because that is the slow part and, applied by a human, is not a CI job at all. It runs on every pull request and it blocks.
The report is the full suite, rubrics included, nightly and on demand. It never blocks. It posts a per-class diff against the last known-good baseline — the table from the previous section — so a human reads what moved and decides.
jobs:
eval-gate: # every pull request, blocks the merge
steps:
- run: pytest evals -m "critical and deterministic"
eval-report: # nightly and on demand, never blocks
steps:
- run: python -m evals.run --all --baseline main
- run: python -m evals.report --post-summaryWhat to gate on, precisely: zero failures in the must-never classes, and no class regressing past a threshold you wrote down in advance. Deciding the threshold after seeing the number is how a gate stops being a gate. And do not gate on a metric whose run-to-run variance you have never measured — a suite that fails one time in five teaches people to re-run it, and after a month nobody reads it at all.
When the suite is too slow, shrink the gate, never the suite. Move cases into the nightly report; run stages in parallel; cache retrieval against a pinned corpus so only generation costs anything. When a case is flaky, resist deleting it. Flakiness usually means the check is too brittle rather than the case being bad — an exact string match on a sentence the model legitimately phrases two ways should be a binary rubric question instead. If you cannot fix it today, quarantine it: keep it running, keep it reported, stop it blocking, and give it an expiry date. A quarantine with no date is a deletion with extra steps.
A golden set decays, and it decays quietly, in three ways.
The first is that you overfit to it. You tune the prompt until the failing cases pass, and after enough rounds the set no longer predicts anything about inputs it has not seen — it has become training data for your prompt. The defence is a holdout: cases you run before a release and do not read individually while iterating. When the holdout starts diverging from your working set, the working set is exhausted and needs new cases, not more tuning.
The second is expectations that encode product decisions which have since changed. When the refund window genuinely changes, the case is now wrong and must be edited — that is normal, and it is a product decision that belongs in the diff with a reason, reviewed by whoever owns the policy.
The third is cases that no longer test anything, because the branch of behaviour they covered was removed. Retire them with a line saying why rather than deleting them silently — the record stops someone re-adding the same case next year, and tells you when a whole failure class has quietly gone unguarded.
One last piece of hygiene: whatever list you keep of known, accepted failures, give every entry a date it gets revisited. An accepted failure with no date is not a decision, it is a lie that ages.
THE SET
curated, not sampled one quota per failure class you fear
every case has a class no class fits? you found a new class
grows from incidents production failure -> case, then fix
watch it fail first red run proves it tests what you think
retire with a reason never delete a case silently
CHECKS
exact where you can schema valid, citation exists, flag set
rubric where you must 3-5 binary questions, all must pass
never a 1-5 score it drifts between runs and between people
version the rubric scores across versions do not compare
agreement check two graders, ten outputs; disagree = bad
question, not bad graders
METRICS
read it per class the overall number hides every trade
compare to a baseline movement is signal, absolute level is not
pair every metric coverage with correctness, escalation
rate with escalation precision
floor of zero must-never classes gate at no failures
STAGES
retrieval right articles in the top k, alone
generation pinned context, so retrieval cannot lie
contract schema, citation ids, required fields
end to end small; the only cross-stage signal
CI
gate fast + deterministic + critical, blocks the merge
report full suite + rubrics, nightly, posts a per-class diff
slow shrink the gate, never the suite
flaky fix the check, quarantine with an expiry date
never gate on a threshold you picked after seeing the number
HONESTY
hold out cases you do not read while tuning
editing an expectation is a product decision, reviewed as one
every accepted failure carries a date it gets revisitedYou now have the shape of an evaluation a team can live with: a set curated around the failures you fear, fed by real incidents, graded by binary questions instead of a scale that wanders, reported per class against a baseline, split by stage so a red run names its own cause, and wired into CI as a small gate plus a large report.
The gap is where those production failures come from in the first place. "Observability for Non-Deterministic Systems", next in this course, answers that: tracing a request through prompts, tools and retries, and logging enough to reproduce a bad answer without keeping what you should not. Every case in your golden set starts as a trace someone was able to find.
Go and do one thing this week. Take the last bug your AI feature shipped, dig out the exact input, and write it up as a case with a class and a provenance line. Run the suite and watch it go red against the prompt you have today. That single red case is worth more than the twenty you would have invented.