Human-in-the-Loop Design
Approval gates, confidence thresholds and escalation paths that fit real work. Designing against reviewer fatigue, where rubber-stamping makes the human a liability rather than a control.
Approval gates, confidence thresholds and escalation paths that fit real work. Designing against reviewer fatigue, where rubber-stamping makes the human a liability rather than a control.
The refund went out at 02:14 on a Tuesday: $8,400 to an account that had been suspended for non-payment two months earlier. The audit log is unambiguous. The agent proposed it, and a human clicked Approve eleven seconds later. The reviewer is not lying when she says she has no memory of it — she approved 380 items that shift and rejected two, which is roughly what she does every shift.
The gate existed. It was staffed, funded, and named in the compliance document. What it was not, that Tuesday or on any of the sixty shifts before it, was a control. By the end of this lesson you will know how to pick the small set of actions worth a human's attention, where to place the gate for each of the three different jobs people ask one to do, how to build a review that can actually be decided in the time it gets, and — the part that decides whether any of the rest works — how to measure whether your reviewers are still looking.
Human-in-the-loop is not a property you switch on. It is a quantity of attention you spend, and the quantity is fixed by something other than your intentions.
Take a support desk where an agent proposes 4,000 actions a day: replies, refunds, credits, plan changes, closures. A reviewer holding real attention — reading the evidence, forming a view, sometimes saying no — manages something like 40 items an hour, for perhaps six productive hours. Three reviewers reviewing everything gives each item 16 seconds. The $8,400 refund gets 16 seconds, and so does the reply that says "thanks, closing this out".
Route 3% of that volume to the same three people and each item gets nine minutes. Same headcount, same salary line, a review that is a different thing entirely.
This is the arithmetic that makes the rest of the lesson follow. Every item you put in the queue takes attention from the items that needed it, and it does not degrade the gate gradually — it degrades it in exact proportion. Reviewing more is how you review worse. A team that decides to "have a human check the AI's work" and means all of it has chosen, without noticing, to have a human check none of it.
The selection rule is one expression: the expected cost of being wrong, which is the probability the action is wrong multiplied by what it costs when it is. The first term comes from your evals. The second is where people stop too early, because the face value of an action is not its cost.
Two multipliers do most of the work. Reversibility is what it costs to undo, including the part that cannot be undone — a sent email, a disclosed fact, a deleted record, a customer who now knows your system got their name wrong. Detectability is whether you find out at all, and how fast. An error the customer reports within the hour is bounded by definition. An error nobody notices has been repeating for as long as it has existed, and its cost is one mistake multiplied by every time the same path ran.
Notice what is absent from that expression: how confident the model was, how long it reasoned, how much text it produced, how sophisticated the output reads. Those track how interesting an item is to a human reading it, not what it costs when wrong. A one-word answer that flips a boolean on a billing record outranks a 900-word analysis nobody will act on.
Bad — fills the queue with the model's hard cases, which are long, ambiguous, and mostly harmless.
def needs_review(action, model):
if model.confidence < 0.85:
return True
if len(model.reasoning) > 2000:
return True
return FalseGood — fills it with the business's expensive cases, priced for undoing and for going unnoticed.
REVIEW_BUDGET_USD = 25.0 # expected loss worth a human minute
def needs_review(action):
p_wrong = measured_error_rate(action.kind) # from evals
cost = action.amount_usd * undo_multiplier(action.kind)
if not action.is_self_reporting:
cost *= 4 # nobody complains, so it repeats
return p_wrong * cost > REVIEW_BUDGET_USDUnder the first rule, the routine credit note the model was confident about goes straight through, every day, at any amount. Confidence is a property of the model and cost is a property of your business — and only one of those changes the week you swap models underneath.
The second rule has a consequence worth stating plainly, because teams flinch from it: it sends most output to nobody. That is the intended result. The reviewed items get attention because the unreviewed ones are not competing for it.
Confidence is not useless — it belongs on a different gate. There are three, they answer three different questions, and almost every muddled review process is two of them wearing one name.
Buys prevention.
Fires on the action's blast radius. The effect is computed, held pending, and executes only on approval.
If it is late, the product stalls — the queue depth is a production dependency. A rejection means the action never happens.
Buys coverage of known-hard cases.
Fires on a signal — a low score, an empty retrieval, an input unlike anything in your evals, two passes that disagreed.
Its volume is unstable, spiking precisely when something upstream changed. A rejection means repair after the fact.
Buys an error rate you can quote.
Fires on a coin flip, after the fact. Prevention is not the point.
If it is late, your dashboard lags. A rejection means one labelled datapoint.
Conflating them shows up in two specific ways. The first is sampling the queue instead of the population: you review a fraction of what was already flagged, then quote the result as your quality number. It describes your hardest 3% and says nothing at all about the 97% that shipped unexamined. The second is leaning on the confidence gate for safety — the model's uncertainty has no reliable relationship to which actions hurt, and an attacker steering your system will be aiming for the confident path anyway.
Two neighbours sit close enough to name once: a filter that blocks output with no human in the path is a guardrail, which Guardrails and Policy Enforcement covers, and an automated score standing in for a human belongs to LLM-as-Judge and Its Failure Modes. Everything here assumes a person makes the call.
Now the review itself. A generated justification is not evidence. It is an argument produced by the same process that chose the action, optimised to read persuasively, and reading it pushes the reviewer toward approval — fluency is convincing, and a reviewer grading an argument has stopped doing the task and started marking an essay.
The test to hold a review item against is whether it is decidable: can this reviewer reach a confident answer from what is on the screen, in the time this item gets? If deciding requires opening the billing console and the ticket history in other tabs, they will not open them. They will approve.
Bad — hands over the model's case for the action, followed by 1,800 more characters in the same register.
{
"item_id": "rv_8817",
"confidence": 0.79,
"action": {
"kind": "refund",
"amount_usd": 8400.0,
"invoice": "INV-2291",
"destination": "card_4417"
},
"model_reasoning": "The customer reports being charged twice…"
}Good — hands over the facts the decision turns on, resolved and on one screen.
{
"item_id": "rv_8817",
"confidence": 0.79,
"action": {
"kind": "refund",
"amount_usd": 8400.0,
"invoice": "INV-2291",
"destination": "card_4417"
},
"evidence": {
"charges": [
{"date": "2026-03-04", "usd": 8400, "state": "settled"},
{"date": "2026-03-11", "usd": 8400, "state": "failed"}
],
"account_state": "suspended_nonpayment",
"prior_refunds_90d": 0,
"policy": "billing/duplicate-charge#2"
}
}Only one of those two charges ever took money. The second is the
retry that failed, which is also why the account is suspended.
The whole item turns on one field, state, and that field appears
nowhere in eighteen hundred words of fluent argument — because the
model never fetched it, and a reviewer reading the argument has no
way to notice an absence.
Keep the reasoning available behind one click. It is useful once the reviewer has formed a view and wants to know how the system got somewhere else. It is corrosive as the first thing they read.
Approve is one click. Reject opens a modal, requires a category, asks for a justification, and triggers a customer notification the reviewer will be asked about later. Nobody designed that as a bias. It arrives one reasonable requirement at a time, and it prices the two outcomes differently in a job where the reviewer is racing a queue.
Four rules keep the scales level.
Symmetric effort. If approve has a keyboard shortcut, reject has one too, with the same number of keystrokes and no dialog in between. Rejection reasons come from a short pick list and are optional — collect them because they are your best signal about what the system is getting wrong, not because the reviewer owes you an essay.
No default. No pre-selected option, no Approve button that happens to catch a stray Enter, no approve-all for the visible page, no ordering that puts approve where the previous screen's cursor already was.
A third outcome. Escalate means "I cannot decide from this", and without it every ambiguous item becomes an approval, because rejecting something probably fine is the error that feels personally expensive to the reviewer. Give it a button and treat its rate as a signal about your review item, not about the person pressing it.
Reversible where possible. A gate whose rejections can be undone in ten seconds is one a tired reviewer will use.
Everything above exists to prevent one outcome, and the mechanism behind it is arithmetic rather than character.
Suppose one item in 400 should be rejected. Then "approve" is the correct answer 99.75% of the time. A reviewer who stops reading entirely scores 99.75% on any accuracy measure you can build, is almost never told they were wrong, and on the rare occasion they are told, hears about it months later with no memory of the item. Sustained attention to a signal that rare degrades within tens of minutes — this is measured, it is worse the rarer the signal, and it cannot be trained or exhorted away. It is a property of the task you built.
The result is rubber-stamping: a queue with a very high approval rate teaches approval, and it teaches it fastest to your most experienced reviewer. At that point the human is supplying accountability without supplying safety. That combination is strictly worse than no gate, because everyone downstream — the on-call engineer triaging an incident, the auditor, the customer, the person deciding whether to widen the agent's permissions — is reasoning from the belief that these actions were checked.
Three design levers work against this, and the first is much stronger than the other two.
Keep the base rate of bad items high. A queue that rejects one item in eight keeps a reviewer awake; one that rejects one in 400 cannot. This is the same lever as the selection rule two sections up — fewer, more expensive items is also the arrangement in which a reviewer regularly finds something. If a class of action is genuinely so reliable that reviewers never reject it, that class does not need a gate; move the gate somewhere it will bite.
Cap the session. Vigilance falls off inside the hour, so rotate people through the queue rather than assigning it for a day.
Never mix stakes in one queue. High-volume trivia sitting alongside rare, expensive items trains the rhythm of the common case onto exactly the item that needed a different one.
Four numbers, and only the last one measures the thing you care about.
Rejection rate, per queue and per reviewer, over time. There is no correct value — there is a band you chose deliberately and an alarm on movement out of it. A falling rate means either the system improved or the reviewer stopped looking, and this number cannot distinguish those two, which is the entire reason for the other three.
Time per item, as a distribution rather than a mean. Watch the tenth percentile and the count of decisions under two seconds. Rubber-stamping has a signature: a run of near-identical very short intervals, usually late in a session.
Agreement between reviewers. Route a fixed fraction of items to two people independently, hide each from the other, and compare the decisions. Correct for chance before you read the result — at a 99% approval rate, two reviewers who approve everything agree 98% of the time, so raw agreement is worthless and you want a chance-adjusted statistic. Falling agreement usually means your criteria are ambiguous rather than your reviewers careless, and the fix is a clearer written rule, ideally one the reviewers helped write.
Seeded probes. Items you constructed to be rejected, inserted into the live queue at a known rate, indistinguishable from real work. The catch rate is the only direct measurement of whether the gate stops anything, and every other number is a proxy for it.
Probes have four rules. They must be incapable of taking effect — intercept them at the executor, keyed on the probe flag, never on the reviewer's decision:
def execute(item, decision):
if item.is_probe:
record_probe_outcome(item.id, decision)
return # never reaches the payment system
if decision != "approve":
return
dispatch(item.action)Reviewers must know probes exist and roughly how often, and must not be able to pick one out; covert testing buys you a worse problem on the day someone works it out. Probe results measure the gate, not the person — the moment catch rate becomes a punishable performance number you get probe-spotting heuristics instead of measurement. And probes must be near-misses, because an absurd one measures nothing; the error you are trying to catch is the plausible one.
Be honest about the resolution you are buying. A dozen probes per reviewer per week will show you a collapse from most-caught to half-caught. It will not resolve a five-point drift, and reporting it as though it does is how a measurement turns back into a comfort.
Escalation needs a named destination and a stated response time. An escalate button that routes into an unmonitored inbox is trained out of use inside a week, and the items that would have gone there become approvals — which is worse than not having offered it.
Reviewers also need a channel that is not per-item. When someone notices that every plan-change proposal for annual accounts has the proration backwards, rejecting a hundred items one at a time is the only vocabulary a review queue gives them, and it is a terrible one. Give them a way to flag a class, and route it to whoever owns the prompt.
Then queue depth, which is a signal and not only a capacity problem. A queue grows for two reasons: volume went up, or the system's behaviour changed. The second is why the queue is longest at precisely the moment the gate matters most — a prompt edit, an upstream schema change, a new attack pattern, and a model swap all surface first as a spike of flagged items.
Bad — clears the backlog by approving whatever nobody reached.
def sweep_expired(queue):
for item in queue.older_than(minutes=30):
execute(item, decision="approve")
item.mark("auto_approved_timeout")Good — clears the backlog by not acting, and makes the backlog somebody's problem.
def sweep_expired(queue):
for item in queue.older_than(minutes=30):
item.mark("expired_unreviewed")
notify_customer_of_delay(item)
if queue.depth() > QUEUE_DEPTH_ALARM:
pause_proposals(kind=queue.kind)
page_on_call("review queue over depth")The first version reaches its highest approval rate on the worst day the system has: the morning a prompt change starts producing subtly wrong refunds is the morning the queue triples, and the sweep waves all of them through. Every one carries a status that a downstream report will count as reviewed.
Two habits follow. Size the gate for your 95th-percentile volume rather than your mean, or the overflow policy is your real policy most weeks. And prefer shedding load at the source: stop the agent proposing actions of that kind and fall back to whatever the process was before it existed. Slower is a cost you can explain to a customer. Wrong is not.
CHOOSING WHAT GETS REVIEWED
expected cost = P(wrong) x cost of being wrong
cost of wrong = amount x hard-to-undo x nobody-notices
never route on confidence alone, output length, how
impressive the output reads
aim for few items, high rejection rate
accept most output is reviewed by nobody
PLACING THE GATE
pre-action before an irreversible effect; blocking;
reject = it never happens
confidence on a signal; volume spikes on change;
not a safety gate
sampling random, drawn before every filter;
measures the 97% that shipped
BUILDING THE REVIEW ITEM
show evidence, resolved action, the policy
clause that decides it
demote the model's reasoning, one click away
bind approval to the exact action payload
never a pre-selected option, a countdown that
approves, an approve-all button
outcomes approve / reject / escalate, equal cost
CATCHING RUBBER-STAMPING
approval rate a band you chose, alarmed on drift
time per item p10, and the count under two seconds
agreement two reviewers, chance-adjusted
seeded probes known-bad, plausible, intercepted before
execution, never a performance metric
sessions rotate inside the hour; never mix stakes
WHEN THE QUEUE IS TOO LONG
size for p95 volume, not the mean
on overflow stop proposing; fall back to the old path
on expiry mark unreviewed; never approve
escalation a named destination with a response time
class feedback one flag, not a hundred rejectionsA gate is worth having when a small number of expensive, hard-to- undo actions get a decidable review from someone who still rejects things — and it is worth removing when it is not, because the version that stays is the one that quietly launders unreviewed actions as approved ones.
The next lesson, The Economics of an AI Product, prices the part this one deliberately left as a fixed budget. Attention has a salary attached, and once you put a number on a reviewed minute and set it against the errors the gate prevents at your volume, the threshold in your routing rule stops being a guess. The argument for reviewing far less gets sharper, not weaker.
The thing to go and do is a measurement. Take the gate you already run and compute one number: its approval rate over the last thirty days. If it is above 98%, you do not currently know whether that queue is a control or a formality. Then build ten probes out of errors your own system has actually made, put them through the queue this week, and count how many come back rejected. That catch rate is the first honest thing your gate has ever told you.