Measuring Instead of Vibing
Your first evaluation: a fixed set of real inputs, an expected outcome for each, and a number you can compare. How to build one in an afternoon and why prompt tuning without it is guessing.
Your first evaluation: a fixed set of real inputs, an expected outcome for each, and a number you can compare. How to build one in an afternoon and why prompt tuning without it is guessing.
A colleague says the summaries your support-triage tool writes are too long. You add "keep it under 20 words" to the prompt, run it on her email, read the result, nod. Ship it.
Two days later someone else asks why the order number stopped appearing in the summaries. It did not stop — it was squeezed out by the word limit you added, on every email long enough to need one. You fixed the input in front of you and quietly broke a hundred you never looked at. By the end of this lesson you will have the smallest thing that would have caught it the same afternoon: twenty real inputs, a written-down expectation for each, and one number you can compare before and after every change you make.
An evaluation — an eval, in practice — is a fixed set of inputs you run every time you change something, with a written expectation for each one and a score at the end. That is the whole idea. The reason it matters more here than in ordinary software is worth being precise about.
When you change a line of ordinary code, you can read the change and reason about what else it touches. A prompt does not work that way. It is not a set of instructions the model executes; it is text that shifts the model's behaviour across every input at once, including inputs whose existence you have forgotten. "Keep it under 20 words" did not mean "trim the long ones" — it meant "when something has to go, drop whatever seems least important," and on a long email the thing that seemed least important was a number.
That is a regression: something that used to work and does not any more. In ordinary code a regression usually breaks loudly. Here it produces a fluent, plausible, slightly worse answer on an input nobody is watching, and it sits there until a person complains.
Editing a prompt while looking at one output is editing a spreadsheet formula while looking at one cell. The cell looks right. You have no idea what happened in the other four hundred rows, because you are not looking at them and nothing turns red.
Start by collecting the inputs. Twenty is a working number, not a scientific one: small enough that you can run them all and read every output in an afternoon, large enough that one lucky answer does not move the score much.
They have to be real. Go to wherever the feature's input actually comes from — the support inbox, your request logs, the ticket queue, the messages people already send you — and take twenty. If the feature is not live yet, take twenty pieces of the real material it will read on its first day.
The temptation is to sit at your desk and imagine twenty inputs. Everyone does it, because it takes four minutes instead of forty, and it produces a set that cannot fail.
Bad — twenty inputs invented at a desk, all well-formed, all the same shape.
1. "Where is my order #10482?"
2. "I would like a refund for order #10483."
3. "My order #10484 arrived damaged."
4. "Can I change the address on order #10485?"
... seventeen more polite sentences containing one order numberGood — twenty pulled from a real inbox, ordinary and awkward mixed.
1. "Where is my order #10482?"
2. "?"
3. forwarded thread, four quoted replies, question in the last
4. "still waiting. third time i've written about this."
5. angry, no order number, no actual question
6. mentions #10483 and #10484, asks about only one of them
7. written in a language your prompt never mentions
8. a two-page complaint with the request in the final line
9. asks about something your company does not sell
10. an auto-reply from a mailbox, no human content at all
... ten ordinary ones, because most real mail is ordinaryThe first set passes on day one and keeps passing forever. It measures the model against a world your users do not live in, so the score stays green while the complaints keep arriving — and a green number you trust is worse than no number at all.
Aim for roughly half ordinary and half awkward. Real traffic is mostly ordinary, but the ordinary cases are the ones already working. You are building this to find the edges, and the edges are where every complaint you have ever received came from.
Each input plus its expectation is one case. The expectation is the part people skip, and skipping it is what turns an eval back into vibing with extra steps.
Write the expectation from the input alone, before you run anything. The moment you read the model's actual output, your sense of what a correct answer looks like bends towards what you are seeing — you will find yourself deciding that a summary without the order number is fine, because the one in front of you does not have it.
An expectation is what the answer must be true of, not the exact words you would have written. Two different good summaries of the same email share almost no wording. They both name the order number.
Bad — an expectation that cannot fail.
input: "Where is my order #10482? Ordered on the 3rd."
expected: a good, helpful summaryGood — an expectation you could hand to someone else and get the same verdict.
input: "Where is my order #10482? Ordered on the 3rd."
expected: category is "shipping"
summary contains "10482"
summary is one sentence, 20 words or fewer
summary does not state a delivery dateThe vague version passes whatever the model does. You read the output, it reads fine, you tick it — and it would have read fine before your change too. A case that cannot fail records your mood on the day you ran it, and detects nothing.
Notice the last line of the good version. "Does not state a delivery date" is there because the model once invented one, and a case is how you make sure it stays fixed.
Split every expectation into two columns.
No argument possible.
The category equals shipping. The summary contains 10482.
The output parses. The summary is 20 words or fewer.
Costs nothing to re-run, never gets tired, and never quietly changes its standards on a Friday afternoon.
One yes/no question, every time.
Does the summary describe what this customer actually wants? Would the colleague who picks up the ticket know what to do from it alone?
Resist scoring out of five — a scale lets you give everything a 3 and move on, which is a way of avoiding an answer rather than giving one.
def check_case_01(result: dict) -> list[str]:
"""Return every failure found. Empty list means it passed."""
failures = []
if result["category"] != "shipping":
failures.append(f"category was {result['category']}")
if "10482" not in result["summary"]:
failures.append("order number missing")
if len(result["summary"].split()) > 20:
failures.append("summary over 20 words")
if "arriv" in result["summary"].lower():
failures.append("summary predicts a delivery")
return failuresWrite the human judgement down as one yes/no question, the same question for every case, and answer it honestly. Real scoring rubrics exist and are useful; they are also a problem for later. The yes/no version fits in an afternoon and catches most of what a rubric would.
The runner is not clever. It walks the cases, calls the same function your product calls, collects the exact-check failures, and prints them.
import json
cases = json.load(open("eval/cases.json"))
results = []
for case in cases:
output = triage_email(case["input"]) # the real function
results.append({
"id": case["id"],
"output": output,
"failures": run_exact_checks(case, output),
})
passed = [r for r in results if not r["failures"]]
print(f"exact checks: {len(passed)}/{len(results)}")
for result in results:
if result["failures"]:
print(result["id"], "->", "; ".join(result["failures"]))exact checks: 16/20
case-04 -> category was other
case-09 -> order number missing
case-13 -> summary over 20 words
case-17 -> category was refund; order number missingThen you do the human column: read the sixteen that survived, answer your one question for each, and count. Say fourteen of them earn a yes. Your pass rate is 14/20 — 70%.
Run each case once. The same input can produce a different answer on a second run, which is how sampling works and not a flaw in your set; if a case flips between runs, note it as a case worth watching rather than averaging it away.
An eval decays in one specific way: the failing cases become annoying, and you make them go away. This happens gradually, always with a good reason at the time, and it is the only thing that can truly kill the value of the set.
Bad — the case is failing, so the case changes.
case-09 expected: summary contains "10483"
actual: "Customer is chasing a late delivery."
action: removed the order-number expectation
result: 17/20Good — the case is failing, so the system changes.
case-09 expected: summary contains "10483"
actual: "Customer is chasing a late delivery."
action: prompt now requires the order number whenever
the email contains one
result: 17/20, and case-09 stays red until it passesBoth rows read 17/20. Only one of them earned it. An evaluation you edit until it agrees with the current output is a record of what your system does, which you already had — the value was always in it disagreeing with you.
Three habits keep the set alive.
Every complaint becomes a case, before the fix
Written down before you attempt it. That way you find out whether your fix worked — and you find out later if it stops working.
Never delete an input for being unrealistic
A real person produced it. "Unrealistic" almost always means "inconvenient".
Version it beside the prompt it measures
And change both in the same commit. A set that lives on someone's laptop stops matching the product within a month.
Everything above exists to make one habit cheap. You change one thing, run the whole set, and write down what happened.
date change exact judged
03-11 baseline, no changes 16/20 14/20
03-11 "always include the order number" 19/20 17/20
03-12 "keep it under 20 words" 19/20 12/20 <--
03-12 loosened to "under 25 words" 19/20 17/20
03-13 swapped in the cheaper, weaker model 17/20 15/20Row three is the bug this lesson opened with. The exact checks barely moved, because the summaries were still short and still contained order numbers — but five of them lost the thing that made them useful, and the human column saw it immediately. Ninety seconds of reading, on the same afternoon, instead of two days and a colleague's irritation.
Row five is why "one thing" is not fussiness. Had you swapped the model and rewritten the prompt in the same step, that two-point drop tells you nothing: you cannot tell whether the cheaper model is adequate, or whether your new wording was worse, or whether both moved and cancelled out. Two separate runs feel slower for about a minute, and save an evening of untangling.
That is the loop, and it is small enough to keep forever: change one thing, re-run the set, compare to the row above.
BUILD IT — one afternoon
collect 20 real inputs # logs, tickets, the actual inbox
~half ordinary # these mostly pass already
~half awkward # empty, angry, long, ambiguous
write expectations first # from the input, before you run
exact where possible # equals, contains, parses, length
one yes/no question else # never a 1-to-5 score
run all 20, every time # never a sample of your own set
record two numbers # exact passed, judged passed
RUN IT — every change
change one thing # prompt OR model OR setting
re-run the whole set # same cases, same checks
compare to the last row # the delta is the only signal
log it # date, change, both numbers
KEEP IT HONEST
complaint becomes a case # same day, before you fix it
failing cases stay failing # fix the system, not the case
never delete an ugly input # a real user produced it
cases are never examples # a shown answer proves nothing
cases live in version control # they ship with the prompt
READ IT RIGHT
70% is not a grade # it is a baseline
70% after 85% is a bug # this is the entire point
100% for months is stale # add the cases that hurtYou can now tell whether a change made things better. That sounds modest and it is the difference between engineering and guessing: twenty real inputs, an expectation written before you looked, a number you compute the same way each time, and a rule that the set changes only when the world does.
Next comes The Shape of an AI Feature, which assembles everything in this course into one small end-to-end thing — input handling, prompt, structure, validation, fallback — and shows you where the eval sits in that picture and which parts of it were never AI at all. Further out, scoring rubrics, metrics chosen to move when quality moves, and a regression suite that runs automatically on every change are the subject of Evaluating an AI Feature in the next course.
Before any of that, do the one concrete thing. Open whatever your AI feature reads in real life, take the first twenty inputs you find — including the three that make you wince — and run today's prompt against them without changing a word of it. Write the number down. You have a baseline now, and from this afternoon on, you will know when you make things worse.