advanced35 minby DevFox

Incidents in Non-Deterministic Systems

Responding when there is no stack trace: reproducing from traces, replaying a request, rolling back a prompt, and writing a postmortem whose root cause is a probability rather than a line of code.

AI
DevOps

Three tickets in two days, all about the same feature, all saying it told a customer something that is not true. You paste each question in yourself and get a correct answer every time. Every dashboard is green: no 5xx, latency flat, no failed jobs, and the on-call phone has not rung once.

Something is wrong and nothing has broken. By the end of this lesson you will be able to run that as an incident — notice it without an alert, narrow it to one of six causes, reproduce it from a trace by replaying the request rather than the question, decide how bad it is when the system is up and confidently wrong, reach for one of five mitigations knowing what each costs, and write a postmortem whose root cause is a probability rather than a line of code.

The incident that never pages

An ordinary incident is detected by the system. A process dies, a queue backs up, an error budget burns, and something wakes somebody. This class of incident is detected by people, because nothing in it throws. A wrong answer is a successful request: a 200, a plausible paragraph, a normal latency.

You already alert on the outcomes the system chose rather than the errors it hit. During an incident those same numbers stop being alert config and become a detector panel — the first thing you open, read as rates against their own trailing baseline, per feature. Each one points somewhere different. Refusal rate up means the model is declining work it used to accept, which usually means what it was given changed, not what it is. Fallback rate up means a dependency is failing and your second path is quietly carrying traffic. Truncation rate up means the output limit is being hit, so something grew — the prompt, the context, the model's appetite for preamble. Empty-retrieval rate up means the corpus or the index, and it is the single most informative number on the panel. Cost per request up means the assembled context grew or a loop is running longer than it should.

Now the uncomfortable part. For the worst failure in this class — fluent, confident and wrong — not one of those moves. A wrong answer and a right answer are the same shape at the level of a rate: same stop reason, same token counts, same latency, same happy 200. There is no machine signal for incorrect. Sampling live traffic into an automated judge closes part of that gap, at the cost of a second system whose own biases you now have to trust.

Which leaves the thing that actually detects these incidents in practice, and it is worth saying without embarrassment: complaint volume. Support tickets, thumbs-down clicks, a screenshot in a sales channel. Most teams treat that as anecdote and their metrics as data, which is backwards here — the anecdote is measuring the failure directly and the metrics are measuring things that correlate with it.

The number to improve afterwards is not how fast you fixed it. It is detection lag — the wall-clock gap between the first affected request, which you reconstruct from traces rather than from memory, and the moment somebody declared an incident. In this class it is routinely measured in days, and every hour of it is more wrong answers already delivered.

One rule makes the difference between a detected incident and a closed ticket: a cluster of complaints about one feature opens an incident whether or not anyone can reproduce it. "Could not reproduce" is the expected outcome here, not a resolution. The next section is why.

Six suspects and what separates them

The instinct is to ask what is wrong with the model. The useful question is what changed, and who was allowed to change it. Six candidates cover almost everything, and each has a cheap discriminating test.

The model. Least likely and first suspected. Your traces record a pinned model id, so this is a string comparison: if the id in the bad traces differs from the id in the good ones, someone moved a pin or an alias moved under you, and you are in an upgrade incident rather than a mystery. If the id is identical and behaviour changed anyway, you are looking at a provider-side serving change, and the only evidence you can produce is your own eval suite re-run today against that same pinned id.

The retrieval corpus. The most common cause and the least owned. Documents get edited by people who do not think of themselves as deploying anything; an index rebuild ran overnight; a chunking change shifted every boundary. Compare the chunk ids and scores in failing traces against passing ones, and compare the index version. A claim the model made that appears verbatim in a chunk is not a hallucination, it is retrieval faithfully surfacing something wrong.

The prompt. Trivial to check if the template version is in the trace, which is exactly why it is there. Include the invisible parts: an interpolated date, a config value that feeds the system prompt, a shared rules file another team owns, a tool description someone improved.

A tool. Its result changed shape or content and the model reported the new thing faithfully. The test is direct — call it now with the arguments recorded in the trace and diff the result against the recorded one.

The input distribution. Nothing in your system changed. The questions did. A campaign brought a different segment, a UI change put the feature in front of new users, a locale opened. Cluster this week's inputs against last month's; the failures are usually a cluster that did not exist before, hitting a gap that was always there.

An attack. The input distribution changed adversarially. The tell is concentration — failures clustered on a few accounts, or on requests whose assembled context contains instructions rather than content. Confirming and responding to that is its own discipline, taught in the lessons on the AI threat model and on guardrails; for triage purposes it is the sixth branch, and you rule it in or out by reading the context that was actually assembled.

That window is the method. Take the last known-good trace and the first known-bad one, both from traces rather than from recollection, and list every change from all six sources between them. Usually one candidate survives. Sometimes two, and the next section tells you which.

Replay the request, not the question

Here is the distinction that makes reproduction work at all. A trace holds two very different things: the user's question, and the assembled request — the exact messages that went to the model, the pinned model id, the sampling parameters, the tool results that were interpolated. Reproducing means sending the second one again. Almost everybody sends the first.

Bad — re-runs the pipeline, so it answers with today's corpus, today's prompt and today's tools.

Good — sends the bytes that were actually sent, to the model that actually served them.

The first version is not a failed reproduction, it is a different experiment. The overnight index rebuild that caused the incident is the reason it passes, and passing is what closes the ticket while the cause stays live and keeps answering customers.

Two properties of the good version matter as much as the replayed input. It runs times samples, because an incident is a rate and so a reproduction is a rate — one clean run proves nothing when the failure happens a third of the time. And it takes overrides, which turns a single trace into a controlled experiment. Vary one field at a time against a predicate that recognises the failure:

That output is the whole diagnosis and it took four minutes. The model is not the variable, the prompt version is not the variable, and one retrieved chunk accounts for all of it. This ablation — replay with exactly one recorded field swapped — is the closest thing to a debugger this stack has. It works because the trace froze the inputs; nothing else you have does.

Some things cannot be replayed, and knowing which in advance is worth more than any tooling. A tool result that was recorded is replayable; a tool you have to call again is not the same tool it was on Tuesday. A chunk id resolves to whatever that chunk says now unless you kept the index version. Anything you logged as post-processed output rather than as the model's raw bytes cannot tell you whether the model or your own parser produced the sentence people are complaining about. The lesson on observability for non-deterministic systems is where those fields are chosen; an incident is where you find out which ones you skipped.

Severity when nothing is down

Outages are honest. They announce themselves, they are bounded in time, and users can tell that the thing is broken and route around it. A confidently wrong answer has none of those properties: it is silent, it is unbounded until someone notices, and users act on it precisely because it does not look broken. For a system that gives advice, quotes numbers or writes to other systems, wrong is a worse state than down, and your severity scale — usually built around availability — will under-call it every time unless you deliberately correct for that.

Four questions size it, and none of them is "how many people complained".

Is the output acted upon? A wrong summary is read and possibly discounted. A wrong number that a person enters into a refund form, or that an agent writes into a ticket, has already changed the world.

Is it reversible? Bounded by whether the artifact can be recalled: a chat message cannot be, a draft can be, a database row depends entirely on whether you kept the previous value.

Does it leave the building? Internal-only wrongness costs trust. Wrongness that reaches customers, regulators or public channels costs more than that, and the difference is often a whole severity level.

Can you enumerate who got one? This is the question nobody asks in the first hour and everybody asks in the sixth. If you can filter traces by feature, time window and the condition you just isolated — say, the runs whose retrieval included that one chunk — an unknown becomes a list of 1,240 requests with user ids attached, and remediation becomes a task with an end. Without it, every severity call is a guess, and guesses under pressure round downwards.

Five levers you can pull in ten minutes

Mitigation does not wait for diagnosis. If the panel says something is badly wrong you pull a lever first and work out why afterwards — the ordering in this lesson is the order of the reasoning, not a requirement to finish triage before you act. Five levers are available immediately, and each costs something specific.

Roll back the prompt
Cheapest and most reversible, and it only exists if templates are versioned and deployable without an application release.
Costs every improvement the newer version carried, given up in one move.
Pin the previous model version
Works only if you kept it pinned, available and evaluated.
Costs you a clock — providers retire versions — and any prompt written against the newer version is now running against an older one.
Disable a tool
Surgical when a tool is the cause, with a failure mode that surprises people: a model does not gracefully lose a capability, it substitutes. Take away retrieval and it answers from its parameters, confidently, with no sources.
Needs a paired instruction saying what to do without it.
Tighten a guardrail
Raise the retrieval threshold, narrow a validator, force a refusal when citations are missing. This is the lever that converts confidently wrong into refuses, which is nearly always the trade you want mid incident.
Costs refusals on questions that were fine — and a feature refusing a quarter of the time reads as an outage.
Turn the feature off
Total, honest, and the only lever that is certainly safe.
Costs everyone it was serving well, plus an organisational cost that is real: a feature switched off during an incident sometimes takes weeks to come back.

Ordered cheapest first. Every one of them costs something, and knowing what beats discovering it at 15:10.

The choosing rule is short. Prefer the smallest lever that makes the failure impossible rather than merely less likely, and prefer levers whose effect you can verify by replay before you declare mitigation.

Bad — a new, unevaluated prompt version shipped into the middle of an incident.

Good — a revert to a version that is already known green, verified before it is called mitigated.

Both took twenty-eight minutes. The first ended with two untested prompt versions, a new user-visible failure and no way to attribute either; the second ended at a state someone had already measured. A forward fix under incident pressure is an unreviewed change to a system you have just proven you do not currently understand.

A root cause that is a rate

The postmortem is where this class of incident is usually written up badly, in one of two directions. The first blames the component: the model hallucinated. The second overcorrects into false precision, naming a single line as the cause because a timeline with a definite answer is more comfortable to present.

Bad — unfalsifiable, and it generates no action anyone can complete.

Good — a mechanism, a measured rate, the change that triggered it, and the part still unknown.

The good version names a rate because a rate is what actually happened — 17 of 20, not "the model said". It states confidence per claim, so a reader can tell evidence from inference. And it records what remains unknown as a finding rather than leaving a gap that a future reader fills in with the comfortable assumption. The bad version cannot be argued with, cannot be disproved, and closes with an action item that no one can mark done — which is how the same incident happens twice with two different postmortems.

Two structural notes on the timeline. Its first row is the first affected request, recovered from traces, not the first complaint — the distance between those two rows is your detection lag, and it is usually the largest number on the page. And every mitigation row should say how it was verified, so that "mitigated" means something a reader can check.

Then the part that makes the incident worth having. Every incident becomes eval cases, and three details decide whether those cases earn their keep.

Capture the assembled context, not the question. The case that would have caught this one is the retrieved set including ch-4471 plus the question — a case built from the question alone runs against whatever the index says next month and will pass for reasons that have nothing to do with the fix.

Assert the behaviour, not the string. "The answer does not claim phone support is available" and "the answer cites a chunk that supports every factual claim" survive a model upgrade and a rewording. An exact-match expectation fails the next time anything improves, and a suite that cries wolf gets muted.

Add the case at the level that would have caught it. This was a retrieval failure, so the primary case belongs in the retrieval tests — does this query surface the current pricing chunk above the retired one — with an end-to-end case as a second layer. An end-to-end assertion alone tells a future engineer that something is wrong and not what.

Finally, add the detector, not only the case. If a person found this incident, the postmortem is incomplete until it names the signal that would have found it first — here, a new-chunk canary that replays a fixed set of queries after every index rebuild and diffs the retrieved ids. An eval suite catches the regression before release. Only a live signal catches the corpus someone edits at 03:10 on a Tuesday.

A cheat sheet to keep

Where to go next

You can now run an incident with no stack trace and no reliable reproduction: read the detector panel, take complaints seriously, intersect the window against all six sources of change, replay the assembled request twenty times with one field swapped, size the damage by whether it was acted on and whether you can enumerate it, pull the smallest lever that bounds the failure, and write up a root cause that is honest about being a probability.

Next comes human-in-the-loop design, which answers the question this lesson can only mitigate after the fact: which outputs should never have reached a customer without a person seeing them first. Approval gates, confidence thresholds and escalation paths are how you stop the highest-severity version of this incident from being possible, and the same lesson covers the way those controls decay when the reviewer starts rubber-stamping.

The thing to do this week takes an hour and tells you whether any of the above is available to you today. Pick one real request from last week's traces, and try to replay it — the assembled messages, the pinned model, the sampling parameters, twenty times. If you cannot, you have just found the first action item of an incident that has not happened yet, and you get to fix it without anyone waiting.