beginner25 minby DevFox

Measuring Instead of Vibing

Your first evaluation: a fixed set of real inputs, an expected outcome for each, and a number you can compare. How to build one in an afternoon and why prompt tuning without it is guessing.

A colleague says the summaries your support-triage tool writes are too long. You add "keep it under 20 words" to the prompt, run it on her email, read the result, nod. Ship it.

Two days later someone else asks why the order number stopped appearing in the summaries. It did not stop — it was squeezed out by the word limit you added, on every email long enough to need one. You fixed the input in front of you and quietly broke a hundred you never looked at. By the end of this lesson you will have the smallest thing that would have caught it the same afternoon: twenty real inputs, a written-down expectation for each, and one number you can compare before and after every change you make.

Why eyeballing is guessing

An evaluation — an eval, in practice — is a fixed set of inputs you run every time you change something, with a written expectation for each one and a score at the end. That is the whole idea. The reason it matters more here than in ordinary software is worth being precise about.

When you change a line of ordinary code, you can read the change and reason about what else it touches. A prompt does not work that way. It is not a set of instructions the model executes; it is text that shifts the model's behaviour across every input at once, including inputs whose existence you have forgotten. "Keep it under 20 words" did not mean "trim the long ones" — it meant "when something has to go, drop whatever seems least important," and on a long email the thing that seemed least important was a number.

That is a regression: something that used to work and does not any more. In ordinary code a regression usually breaks loudly. Here it produces a fluent, plausible, slightly worse answer on an input nobody is watching, and it sits there until a person complains.

Editing a prompt while looking at one output is editing a spreadsheet formula while looking at one cell. The cell looks right. You have no idea what happened in the other four hundred rows, because you are not looking at them and nothing turns red.

Twenty real inputs, and the awkward ones

Start by collecting the inputs. Twenty is a working number, not a scientific one: small enough that you can run them all and read every output in an afternoon, large enough that one lucky answer does not move the score much.

They have to be real. Go to wherever the feature's input actually comes from — the support inbox, your request logs, the ticket queue, the messages people already send you — and take twenty. If the feature is not live yet, take twenty pieces of the real material it will read on its first day.

The temptation is to sit at your desk and imagine twenty inputs. Everyone does it, because it takes four minutes instead of forty, and it produces a set that cannot fail.

Bad — twenty inputs invented at a desk, all well-formed, all the same shape.

Good — twenty pulled from a real inbox, ordinary and awkward mixed.

The first set passes on day one and keeps passing forever. It measures the model against a world your users do not live in, so the score stays green while the complaints keep arriving — and a green number you trust is worse than no number at all.

Aim for roughly half ordinary and half awkward. Real traffic is mostly ordinary, but the ordinary cases are the ones already working. You are building this to find the edges, and the edges are where every complaint you have ever received came from.

Writing down what correct means

Each input plus its expectation is one case. The expectation is the part people skip, and skipping it is what turns an eval back into vibing with extra steps.

Write the expectation from the input alone, before you run anything. The moment you read the model's actual output, your sense of what a correct answer looks like bends towards what you are seeing — you will find yourself deciding that a summary without the order number is fine, because the one in front of you does not have it.

An expectation is what the answer must be true of, not the exact words you would have written. Two different good summaries of the same email share almost no wording. They both name the order number.

Bad — an expectation that cannot fail.

Good — an expectation you could hand to someone else and get the same verdict.

The vague version passes whatever the model does. You read the output, it reads fine, you tick it — and it would have read fine before your change too. A case that cannot fail records your mood on the day you ran it, and detects nothing.

Notice the last line of the good version. "Does not state a delivery date" is there because the model once invented one, and a case is how you make sure it stays fixed.

What a computer checks, and what you check

Split every expectation into two columns.

What a machine can decide

No argument possible.

The category equals shipping. The summary contains 10482. The output parses. The summary is 20 words or fewer.

Costs nothing to re-run, never gets tired, and never quietly changes its standards on a Friday afternoon.

What only a person can decide

One yes/no question, every time.

Does the summary describe what this customer actually wants? Would the colleague who picks up the ticket know what to do from it alone?

Resist scoring out of five — a scale lets you give everything a 3 and move on, which is a way of avoiding an answer rather than giving one.

Push as much as you possibly can into the left one. Your attention is the scarce resource here.

Write the human judgement down as one yes/no question, the same question for every case, and answer it honestly. Real scoring rubrics exist and are useful; they are also a problem for later. The yes/no version fits in an afternoon and catches most of what a rubric would.

Running the set and getting a number

The runner is not clever. It walks the cases, calls the same function your product calls, collects the exact-check failures, and prints them.

Then you do the human column: read the sixteen that survived, answer your one question for each, and count. Say fourteen of them earn a yes. Your pass rate is 14/20 — 70%.

Run each case once. The same input can produce a different answer on a second run, which is how sampling works and not a flaw in your set; if a case flips between runs, note it as a case worth watching rather than averaging it away.

Keeping the set honest

An eval decays in one specific way: the failing cases become annoying, and you make them go away. This happens gradually, always with a good reason at the time, and it is the only thing that can truly kill the value of the set.

Bad — the case is failing, so the case changes.

Good — the case is failing, so the system changes.

Both rows read 17/20. Only one of them earned it. An evaluation you edit until it agrees with the current output is a record of what your system does, which you already had — the value was always in it disagreeing with you.

Three habits keep the set alive.

Every complaint becomes a case, before the fix
Written down before you attempt it. That way you find out whether your fix worked — and you find out later if it stops working.
Never delete an input for being unrealistic
A real person produced it. "Unrealistic" almost always means "inconvenient".
Version it beside the prompt it measures
And change both in the same commit. A set that lives on someone's laptop stops matching the product within a month.

A set that agrees with your product is a set that has stopped measuring it

Change one thing, re-run, compare

Everything above exists to make one habit cheap. You change one thing, run the whole set, and write down what happened.

Row three is the bug this lesson opened with. The exact checks barely moved, because the summaries were still short and still contained order numbers — but five of them lost the thing that made them useful, and the human column saw it immediately. Ninety seconds of reading, on the same afternoon, instead of two days and a colleague's irritation.

Row five is why "one thing" is not fussiness. Had you swapped the model and rewritten the prompt in the same step, that two-point drop tells you nothing: you cannot tell whether the cheaper model is adequate, or whether your new wording was worse, or whether both moved and cancelled out. Two separate runs feel slower for about a minute, and save an evening of untangling.

That is the loop, and it is small enough to keep forever: change one thing, re-run the set, compare to the row above.

A cheat sheet to keep

Where to go next

You can now tell whether a change made things better. That sounds modest and it is the difference between engineering and guessing: twenty real inputs, an expectation written before you looked, a number you compute the same way each time, and a rule that the set changes only when the world does.

Next comes The Shape of an AI Feature, which assembles everything in this course into one small end-to-end thing — input handling, prompt, structure, validation, fallback — and shows you where the eval sits in that picture and which parts of it were never AI at all. Further out, scoring rubrics, metrics chosen to move when quality moves, and a regression suite that runs automatically on every change are the subject of Evaluating an AI Feature in the next course.

Before any of that, do the one concrete thing. Open whatever your AI feature reads in real life, take the first twenty inputs you find — including the three that make you wince — and run today's prompt against them without changing a word of it. Write the number down. You have a baseline now, and from this afternoon on, you will know when you make things worse.