intermediate30 minby DevFox

Evaluating an AI Feature

Golden sets, rubrics, and regression suites that run in CI. Choosing metrics that move when quality moves, and keeping an evaluation honest as the feature it measures changes.

AI
Testing

Someone tightens the system prompt on a Friday afternoon. The twenty cases in your spreadsheet still pass — nineteen of them, the same nineteen as last week — so it ships. On Monday, support has four tickets about the assistant confidently quoting a refund window that was retired two years ago.

The spreadsheet was right about the method and wrong about the scale. Twenty cases and a pass rate got you off vibes, which is the whole point of "Measuring Instead of Vibing" in the beginner course. This lesson turns that afternoon's work into something a team runs on every change: a curated golden set that grows from real failures, rubrics that do not drift, metrics that move when quality moves, per-stage evaluation that tells you what broke, and a CI setup that blocks the merges worth blocking.

What changes when it stops being a spreadsheet

A spreadsheet has one reader and one moment. An eval suite has to survive being run by someone who did not write it, six months later, against a feature that changed underneath it. Three properties get you there, and none are about scoring. It lives in the repository next to the code it measures, so a change to expected behaviour shows up in a diff and gets reviewed. It runs unattended, which means every input it needs is pinned rather than assumed. And each case carries its provenance, so the next person can tell whether it guards something real or was somebody's idea of a nice example in 2024.

That makes a case a file, not a row. Here is one from a help-desk assistant that answers billing questions from a help centre and returns structured output:

yaml

# evals/cases/refund-after-cancellation.yaml
id: refund-after-cancellation
category: stale-policy        # the failure class it guards
added: 2026-03-14
provenance: ticket 88213 — assistant quoted a retired policy
input:
  question: "I cancelled Pro yesterday. Can I still get a refund?"
  account:
    plan: pro
    cancelled_days_ago: 1
corpus_version: help-centre@2026-03-10   # pinned, so it repeats
checks:
  - kind: cites
    article: refunds-and-cancellations
  - kind: must_not_contain
    text: "30-day"            # the retired window
  - kind: rubric
    id: answer-quality

The category and provenance lines are the ones people skip and the ones that make the set maintainable: a case with no class cannot be reported on, and a case with no history cannot be retired with confidence.

A golden set is curated, not collected

A golden set is the fixed collection of inputs your feature is evaluated against, each with an expected outcome. The instinct when growing one is to reach for production traffic and take a big sample. That instinct produces a set that looks impressive and tests almost nothing.

Bad — samples traffic, so the set inherits traffic's shape.

python

recent = load_production_requests(days=30)
cases = random.sample(recent, 200)

Good — a quota per failure class, filled on purpose.

python

QUOTAS = {
    "stale-policy": 12,     # answers from retired documentation
    "must-escalate": 15,    # money, legal, or an angry customer
    "ambiguous-plan": 10,   # question that spans two plans
    "out-of-scope": 10,     # should refuse, not improvise
    "hostile-input": 8,     # instructions hidden in the question
    "happy-path": 15,       # the boring majority, still checked
}
cases = [case
         for klass, limit in QUOTAS.items()
         for case in pick_reviewed(recent, klass=klass, n=limit)]

Real traffic is overwhelmingly easy. The cases that cost you money are rare by definition, so in a random two hundred they show up once or twice; a regression that breaks all of them moves the overall pass rate by half a point, and nobody looks twice at half a point.

The reframe that makes curation obvious: your suite is a specification of behaviour, not a survey of users. You are not estimating how often the feature is right in the wild — that is what production monitoring is for — you are asserting that specific things remain true. So over-represent everything you are afraid of: irreversible actions, the boundary between answering and refusing, inputs that are ambiguous even to a human, inputs actively trying to break you.

Every production failure becomes a case

Curated does not mean invented. The best cases are ones reality handed you, and the discipline is the same one you already apply to bugs in ordinary code: a failure that reached users gets a regression case before it gets a fix.

Three rules keep that from degenerating. Freeze the input exactly as it arrived, including account state and corpus version — a paraphrase is a different question, and models are sensitive to the difference. Record what actually happened alongside what should have happened; the wrong answer is evidence, and later it explains why the case exists. And assign it a class. If it fits none of them, you have found a new failure class, which is worth more than the case itself.

The set grows by failures and shrinks by retirement — never by someone adding a few more examples on a quiet afternoon. Those cases have no class, no provenance, and nothing to say when they fail.

Rubrics: binary questions, not scales

Plenty of checks are exact. Did the response validate against the schema, does the cited article id exist, was the escalation flag set. Those need no judgement and should never be graded by a human or a model.

What is left is the part where judgement is unavoidable: is this answer actually useful to the person who asked. A rubric is how you make that judgement repeatable, and the shape of the rubric decides whether it is.

Bad — one number for the whole answer, so nothing is ever comparable twice.

Good — three independent questions, each answerable yes or no from the output alone.

One score out of five

Compresses, then drifts.

Several independent judgements collapse into one number, so an answer that gained a citation and lost the outcome scores the same as one that did the reverse. The trade is invisible.

And the scale moves: a weak 4 to one grader is a strong 3 to the next, and your own 4 in March is a 3 in June.

Three yes-or-no questions

Stays comparable.

Each answerable from the output alone, and each failure names which property was missing.

Binary questions drift far less, because there is nowhere to drift to.

Same answers, same graders. One of these produces numbers you can still compare in six months.

Three rules follow. Keep it to three to five questions — past that, graders skim. Make each answerable from the output alone, without holding an ideal answer in mind, since "is it as good as it could be" is not a question two people answer the same way. And version the rubric: the moment you reword a question, scores from before and after stop being comparable, so bump a number and say so rather than quietly breaking your own history.

Someone has to apply it. Early on that is a person, and a few dozen cases with three questions each is genuinely an hour's work. Handing the reading to a model is the obvious scaling move and brings its own distortions — the advanced course covers those in "LLM-as-Judge and Its Failure Modes". Either way, sanity-check the rubric on humans first: give the same ten outputs to two people and compare. If they disagree on a question, the question is wrong, not the people.

Metrics that move when quality moves

Because your set is curated, its absolute pass rate means very little — a set full of hard cases scores worse than a set full of easy ones, and neither number describes your users' experience. What matters is movement, and movement per class matters more than movement overall.

Bad — one average, so any improvement pays for any failure.

Good — a floor on the classes that must never break, and no class allowed to slide.

The average is the problem. Fifteen easy cases improving buys exactly enough headroom to hide the one case where the assistant told a customer they were owed money they were not, and the build stays green while the worst thing your feature can do starts happening again.

Reporting per class is what makes a regression legible:

The overall line moved by one case. Underneath it, the change made refusals better and stale-policy answers worse — a real trade someone has to decide about, which the summary number would have shrugged off as noise.

Now the harder version: the metric that improves while users complain. A metric is a proxy, and optimising a proxy is not optimising the thing. You loosen the escalation threshold, so auto-resolution climbs and the escalation rate falls — both look like wins until you notice the escalations you stopped sending were the ones that needed sending. You add "be concise" and the conciseness question starts passing everywhere, because the model dropped the caveat that made the answer correct.

The defence is a counter-metric: every metric that can be gamed by doing more of something gets paired with one that punishes doing it badly. Coverage pairs with correctness. Escalation rate pairs with escalation precision — of the ones you escalated, how many needed it. Conciseness pairs with the question about naming the condition.

Evaluate the parts, not only the whole

An end-to-end score tells you something broke. It does not tell you what, and on a feature with retrieval, generation and a response contract, "what" has at least three answers that need completely different fixes.

So evaluate the stages separately, with the boundary between them pinned. Component evaluation means each stage is measured against fixed inputs, so a failure can only be caused by that stage:

Read the three together and the diagnosis is immediate. Retrieval red, generation green: the model answered well from what it was given, and the articles it needed never arrived — fixing that is the subject of "Retrieval That Actually Retrieves" earlier in this course. Retrieval green, generation red: the material was there and the answer still went wrong, so the problem is the prompt or the model. Contract red alone: nothing is wrong with the thinking, your parser is about to throw, and that is the cheapest failure here to fix.

Keep a small end-to-end suite as well. Stage tests use pinned contexts, so they are blind to the bugs that only appear once the real stages are wired together — a retriever returning three articles where the prompt expects one, a formatter truncating a citation list. Ten end-to-end cases across your riskiest classes catch that, and they are the ones you run last.

Running it in CI without blocking every merge

The suite that catches everything is too slow and expensive to run on every push; the suite that runs on every push catches less than you want. Split it in two, because gating and reporting are genuinely different jobs.

The gate is fast, deterministic and small. Exact checks only — schema validity, citation ids, the escalation flag, the must-never classes. No rubric grading, because that is the slow part and, applied by a human, is not a CI job at all. It runs on every pull request and it blocks.

The report is the full suite, rubrics included, nightly and on demand. It never blocks. It posts a per-class diff against the last known-good baseline — the table from the previous section — so a human reads what moved and decides.

What to gate on, precisely: zero failures in the must-never classes, and no class regressing past a threshold you wrote down in advance. Deciding the threshold after seeing the number is how a gate stops being a gate. And do not gate on a metric whose run-to-run variance you have never measured — a suite that fails one time in five teaches people to re-run it, and after a month nobody reads it at all.

When the suite is too slow, shrink the gate, never the suite. Move cases into the nightly report; run stages in parallel; cache retrieval against a pinned corpus so only generation costs anything. When a case is flaky, resist deleting it. Flakiness usually means the check is too brittle rather than the case being bad — an exact string match on a sentence the model legitimately phrases two ways should be a binary rubric question instead. If you cannot fix it today, quarantine it: keep it running, keep it reported, stop it blocking, and give it an expiry date. A quarantine with no date is a deletion with extra steps.

Keeping the set honest as the feature changes

A golden set decays, and it decays quietly, in three ways.

The first is that you overfit to it. You tune the prompt until the failing cases pass, and after enough rounds the set no longer predicts anything about inputs it has not seen — it has become training data for your prompt. The defence is a holdout: cases you run before a release and do not read individually while iterating. When the holdout starts diverging from your working set, the working set is exhausted and needs new cases, not more tuning.

The second is expectations that encode product decisions which have since changed. When the refund window genuinely changes, the case is now wrong and must be edited — that is normal, and it is a product decision that belongs in the diff with a reason, reviewed by whoever owns the policy.

The third is cases that no longer test anything, because the branch of behaviour they covered was removed. Retire them with a line saying why rather than deleting them silently — the record stops someone re-adding the same case next year, and tells you when a whole failure class has quietly gone unguarded.

One last piece of hygiene: whatever list you keep of known, accepted failures, give every entry a date it gets revisited. An accepted failure with no date is not a decision, it is a lie that ages.

A cheat sheet to keep

Where to go next

You now have the shape of an evaluation a team can live with: a set curated around the failures you fear, fed by real incidents, graded by binary questions instead of a scale that wanders, reported per class against a baseline, split by stage so a red run names its own cause, and wired into CI as a small gate plus a large report.

The gap is where those production failures come from in the first place. "Observability for Non-Deterministic Systems", next in this course, answers that: tracing a request through prompts, tools and retries, and logging enough to reproduce a bad answer without keeping what you should not. Every case in your golden set starts as a trace someone was able to find.

Go and do one thing this week. Take the last bug your AI feature shipped, dig out the exact input, and write it up as a case with a class and a provenance line. Run the suite and watch it go red against the prompt you have today. That single red case is worth more than the twenty you would have invented.