Drift, Regression, and Model Upgrades
Why a better model breaks your prompts, how to pin and how to move, shadow traffic, and an upgrade playbook that turns a scary migration into a measured one.
Why a better model breaks your prompts, how to pin and how to move, shadow traffic, and an upgrade playbook that turns a scary migration into a measured one.
The new model is better. The provider's post says so, the public benchmarks say so, and your own five-minute smoke test says so. You point the invoice extractor at it on a Tuesday morning. By Thursday, field-level accuracy is down four points, the support assistant has become chatty enough that users are complaining, and the one thing everybody agrees on is that the model got better.
Both of those are true at once, and the reason is that you never deployed a model — you deployed a prompt fitted to a model. By the end of this lesson you will be able to tell apart the four unrelated things people call drift, pin a version and say precisely what the pin costs you, and run an upgrade as four measured gates instead of a deploy and a held breath.
Read a system prompt that has been in production for a year and you are reading scar tissue. "Do not open with 'I'd be happy to help'" is there because a model did. "Return only the JSON object, with no prose and no code fence" is there because a model wrapped it. Three few-shot examples showing a two-sentence summary are there because a model wrote six paragraphs. "You may call the order lookup more than once" is there because a model called it once and guessed the rest.
Every one of those lines is a correction: a counterweight aimed at one specific model's specific defect. A correction is a prescription lens. Ground for the eye it was measured on, it produces clear sight; put the same lens in front of a healthy eye and it is the astigmatism. Nothing about the glass changed.
That is the whole mechanism, and it shows up in five recognisable ways.
Overcorrection. The instruction that dragged a rambling model back to four sentences now truncates a model that would have given you the right six.
Few-shot anchoring. Examples are the strongest instruction in a prompt, because they demonstrate rather than describe. A stronger model reproduces your examples faithfully — including their ceiling. Your example set silently caps the new model at the old model's quality.
Verbosity and formatting. Output length shifts, headings and bullets appear where prose used to be, and the downstream regex, the character-limited UI slot, and the token bill all move with it.
The refusal boundary. It moves in both directions. Categories that used to pass now get declined; categories you relied on being refused now sail through, which matters more, because your guardrails were sized against the old boundary.
Tool-calling eagerness. The same tool schemas produce a different number of calls. A model that now checks before answering makes your loops longer and your latency budget wrong; a model that now answers without checking makes them confidently stale.
Bad — a list of one model's habits, which the next model does not have.
Do not begin your reply with a pleasantry.
Do not apologise for limitations.
Do not wrap the JSON in a code fence.
Never write a summary longer than four sentences.Good — the same requirements stated as the output contract they always were.
Output: a single JSON object matching the schema below.
`summary`: at most four sentences, no greeting, no apology.A prohibition only means something against the behaviour it prohibits. When that behaviour is gone, the first version keeps paying a cost — tokens, attention, and an occasional overcorrection — for a defect that no longer exists, and you have no way to tell which lines are still earning their place. The second version describes the output you need, so it survives the model change with nothing to audit.
"The model is drifting" is four claims wearing one word, and the fix for each is different. Three of the four are not fixed by changing models at all.
Self drift — you changed something
A prompt edit three weeks ago, a tool description reworded, sixty new documents indexed, a schema field made optional.
The most common, and the only one you caused.
Version drift — the snapshot moved
You asked for "the latest model in this family", and the provider answered a different question this week than last.
Input drift — your traffic changed
Model and prompt identical. A new market brought a new language, a campaign brought shorter and vaguer questions, one customer started uploading scans instead of digital PDFs.
World drift — the answers went stale
Retrieval returns the same document it always returned. The document now describes last year's pricing.
Telling them apart is a matter of freezing everything except the variable under test, and running the suite you already trust.
frozen = load_suite("golden/2026-05") # cases + expected, unchanged
# 1. control: last month's prompt, pinned model
run(frozen, model=PINNED, prompt=prompt_at("2026-05-01"))
# 2. self drift: today's prompt, same pinned model
run(frozen, model=PINNED, prompt=current_prompt())
# 3. version drift: same prompt, floating alias instead
run(frozen, model=FLOATING_ALIAS, prompt=current_prompt())
# 4. input drift: today's traffic, not the golden cases
run(sample_recent_traffic(days=7), model=PINNED,
prompt=current_prompt())Run 1 failing means the suite or the scorer moved, and you are measuring your own instruments. Run 2 failing while 1 passes is self drift: bisect your prompt and corpus history, because something you shipped did this. Run 3 failing while 2 passes is version drift, and it is the only one of the four where the model is the culprit. Runs 1 through 3 green while 4 fails is input drift — the suite no longer resembles production, and the fix is new cases, not a new model. All four green while users still report wrong answers is world drift: the system is faithfully retrieving stale truth.
A floating alias resolves to whatever the provider currently considers the newest model in a family. It is a convenience that hands a third party a deploy button on your production system, and it fires on their schedule, into whatever state your prompts are in that morning.
Pin instead. A pinned version is an exact, dated snapshot identifier, held in configuration in one place, per role — and roles matter, because they move at different speeds and for different reasons.
models:
serving: reasoning-model@2026-04-17 # what answers users
judge: reasoning-model@2025-11-02 # graded evals; move rarely
embedding: embedding-model@v3 # moving this reindexes all
candidate: reasoning-model@2026-09-02 # what the gates testThe embedding model is the sharpest of the three. Vectors from two different embedding models are not comparable, so changing that line is not an upgrade — it is a full reindex plus a retrieval-quality re-evaluation, and doing it by accident leaves you with an index that half-answers.
Now the part teams skip. A pin is a lease, not ownership. Snapshots are deprecated on the provider's schedule with a retirement window measured in months, and that window opens whenever it opens. Pinning does not remove the migration; it moves the migration off the provider's calendar and onto yours — but only if you actually put it on yours.
A pinned system with no upgrade practice is the worst of both. It runs unchanged for eighteen months, accumulating corrections fitted to a model that is about to stop existing, and then does its first migration in a fortnight against a deadline it did not set. Pin the version, then schedule the playbook below against the newest candidate on a cadence you choose — quarterly is plenty. The point is that the upgrade is never the first time.
Run your evaluation suite through both models paired: the same case, the same seed, the same retrieved context, scored per case, and compared case by case rather than as two independent averages. Pairing removes the case-to-case variance that otherwise swamps the effect you are looking for. The sample size that needs, and what counts as a real difference rather than noise, is the subject of this course's opening lesson on eval harnesses — treat paired comparison here as a tool you already own.
What this lesson adds is the slicing. An aggregate score is a mean over a traffic mix you did not choose, and a model upgrade is the single change most likely to move different parts of that mix in opposite directions.
Bad — two means, so a class that collapsed is cancelled out by classes that improved.
old = mean(score(run(case, MODEL_OLD)) for case in suite)
new = mean(score(run(case, MODEL_NEW)) for case in suite)
print(f"overall {new - old:+.3f}")Good — pairs each case, then reports the movement per class of case.
deltas = defaultdict(list)
for case in suite:
old = score(run(case, MODEL_OLD, seed=case.seed))
new = score(run(case, MODEL_NEW, seed=case.seed))
deltas[case.case_class].append(new - old)
for case_class, values in sorted(deltas.items()):
moved = mean(values)
flag = "REGRESSION" if moved < -0.05 else ""
print(f"{case_class:28} {moved:+.3f} n={len(values):3} {flag}")The first version prints overall +0.012 and you ship. The
second prints the same +0.012 and also prints that
German-language refund cases fell 0.21 across forty examples,
which is the week you were about to spend diagnosing a mystery
after release instead of the ten minutes you are spending now.
Slice by the dimensions you already record: language, document type, tool-using versus direct-answer, adversarial versus benign, long input versus short. Set a per-slice regression floor and let it block the upgrade regardless of the headline. Accepting a slice regression is legitimate — sometimes the class is being deprecated, sometimes the trade is worth it — but it must be a decision somebody writes down, not an average that hid it.
One caution while you read the failures: when the candidate fails a case, check the case before you blame the model. A stronger model routinely exposes golden answers that were wrong or stale all along, and scoring it against them punishes it for being right.
Your suite contains what you thought to write down. Shadow traffic is how you test against what actually arrives: a sample of real production requests is sent to the candidate as well as the incumbent, only the incumbent's answer is served, and both are logged.
You have no labels for this traffic, so you are not measuring correctness. You are comparing distributions — output length, refusal rate, tool calls per request, schema-validation failures, latency, cost per request — and diffing the pairs. Where the two answers differ materially, you have found your next evaluation cases; sample a few dozen of those for human labelling and they become permanent suite members.
Sample rather than mirror everything: shadowing all traffic doubles the model bill for that feature, and two to five percent for a week usually settles the distributions. Shadow outputs land in the same trace store as production outputs, under the same retention and redaction rules — a shadow log is production data wearing a different label.
Bad — the candidate runs against the live tool layer, so its calls reach the world.
def handle(request):
answer = agent.run(request, model=INCUMBENT, tools=TOOLS)
shadow.submit(agent.run, request, model=CANDIDATE,
tools=TOOLS)
return answerGood — the candidate gets a read-only tool layer, so a write becomes a recorded intent.
def handle(request):
answer = agent.run(request, model=INCUMBENT, tools=TOOLS)
shadow.submit(agent.run, request, model=CANDIDATE,
tools=read_only(TOOLS))
return answerShadowing is safe precisely because nothing it does reaches the
world; hand it the live tools and it is not a shadow, it is a
second uncontrolled production run issuing real refunds. And the
write path is exactly where a change in tool-calling eagerness
surfaces first — with read_only, "the candidate tried to issue
three times as many refunds" is a metric you read on Wednesday
rather than a reconciliation you run in January.
Ramp the candidate as a percentage read from configuration at request time. Rollback is then a config write: seconds, no build, no pipeline, no engineer who happens to know how the deploy works. Assume the regression will be spotted by someone reading tickets on a Sunday afternoon, and design the undo for that person.
Three details are specific to a model change rather than to rollouts in general.
Every trace, log line and metric carries the model version as a dimension. Without it your dashboards average two different populations and show you a flat line during the exact window you most need resolution. This is the most common way a careful rollout still learns nothing.
Ramp by a stable hash of the conversation or user, not per request. A user who is served alternating models across turns of one thread gets an incoherent conversation, and your comparison is polluted by a cohort that is neither.
Keep the incumbent fully callable — its prompt variant included — until the candidate has held one hundred percent for a stated period. Deleting the old prompt in the same change that ramps the new model converts a config-write rollback back into a deploy, which is the thing this gate exists to avoid.
Gate one will often show the candidate losing. That result is about the pair, not the model: your deployable unit is the triple of prompt, tool schemas and model version, and evaluating one member of it while holding the others fixed at values fitted to a different model tells you almost nothing.
So re-fit before you judge. Ablate the prompt against the candidate, one named block at a time.
blocks = load_prompt_blocks("prompts/extract.md") # named sections
baseline = suite_score(CANDIDATE, render(blocks))
for name in list(blocks):
trimmed = {k: v for k, v in blocks.items() if k != name}
trimmed_score = suite_score(CANDIDATE, render(trimmed))
if trimmed_score >= baseline - NOISE_FLOOR:
print(f"droppable {name:26} {trimmed_score - baseline:+.3f}")NOISE_FLOOR is the run-to-run variance you measured when you
built the harness, not a tolerance you invent to make blocks
droppable. A block whose removal costs nothing measurable was
carrying nothing, and it was almost certainly a correction for a
model you no longer run.
Give the few-shot examples their own pass, because they behave differently from instructions. Try the suite with five examples, with two, and with none plus a sharper output spec. A stronger model frequently scores best on the shortest set, since the examples were demonstrating a standard it already exceeds. Do not predict which way it goes; the ablation answers in an afternoon and your intuition does not.
Then re-check everything downstream of the output. The parser, the truncation logic, the UI that assumed three bullets — those were written against the old model's output distribution, and looking at one good sample from the new one is not a test.
Keep both prompt variants in the repository, versioned alongside the model they were fitted to, and keep both under evaluation until the old model is retired. A directory is cheap. Discovering during a rollback that the old prompt was deleted is not.
Everything above runs when you decide to change something. Version drift and input drift arrive on mornings when you changed nothing, and an evaluation suite that runs on merge will not see them for weeks.
Two continuous mechanisms cover that gap, and neither is an eval score. The first is the set of distribution signals you already collect — refusal rate, fallback rate, truncation rate, repair rate, output length, tool calls per request, cost per request — each compared against its own trailing baseline, and each segmented by model version. Collecting them is the subject of the observability lesson in the previous course; what belongs here is what a step change in them means. A jump on a day you shipped nothing is version drift or input drift, and the input-side histograms — language mix, input length, document type — tell you which of the two in about a minute.
The second is a scheduled canary: a small fixed set of cases, twenty or thirty, run every few hours against the production configuration rather than a test rig, and scored. It is cheap, boring, and it dates the change. "Something got worse recently" and "it changed at 14:00 on Tuesday, and here is everything else that happened at 14:00 on Tuesday" are different investigations, and only one of them finishes.
TRIAGE — freeze everything but one variable, run the suite
old prompt + pinned model fails ... your suite or scorer moved
new prompt + pinned model fails ... self drift: prompt/tools/corpus
only the floating alias fails ..... version drift: the alias moved
suite green, production bad ....... input drift: traffic != suite
all green, answers stale .......... world drift: corpus is old
three of the four are not fixed by changing models
PINNING
serving / judge / embedding ....... pin each role separately
judge model ....................... never floats; re-baseline on change
embedding model ................... changing it = full reindex
a pin is a lease .................. deprecation is on their calendar
schedule the playbook quarterly ... so the upgrade is never the first
UPGRADE GATES — in order, each can stop the release
1 paired eval ..................... same case, same seed, both models
slice by class .................. language, doc type, tools, length
per-slice regression floor ...... blocks regardless of the headline
candidate "fails" a case ........ check the case before the model
2 shadow traffic .................. 2-5% for a week, serve incumbent
read-only tool layer ............ writes recorded, never sent
compare distributions ........... length, refusals, tool calls, cost
material diffs .................. become new labelled eval cases
3 flagged ramp .................... percentage read at request time
rollback = config write ......... no build, no deploy, no pipeline
model version on every trace .... or the dashboards average it away
hash by conversation ............ never per request
keep the incumbent callable ..... old prompt stays until retirement
RE-TUNE, DO NOT PORT
deployable unit ................... (prompt, tool schemas, version)
ablate block by block ............. drop what costs < the noise floor
few-shot gets its own pass ........ 5 / 2 / 0 examples, measured
re-check the parser and the UI .... output shape moved with the model
keep both variants in the repo .... versioned with their model
WATCH CONTINUOUSLY
distribution signals .............. segmented by model version
step change, nothing shipped ...... version drift or input drift
input-side histograms ............. tell you which of those two
scheduled canary .................. dates the change to an hour
never alert on exact equality ..... sampling makes it fire foreverYou can now separate four failures that share a name, hold a version deliberately instead of by accident, and put an upgrade through gates that each answer a different question: does it win per case class, does it behave on real traffic, and can you undo it in ten seconds.
The gates are what you run before serving. Incidents in Non-Deterministic Systems is the neighbouring question: the regression is already live, users are already affected, and there is no stack trace to read. It covers reproducing a bad answer from its trace, replaying a single request, rolling a prompt back while the clock runs, and writing a postmortem whose root cause is a probability rather than a line of code. Everything here is the preparation that makes those incidents rarer and shorter.
The concrete thing to do this week: take the newest candidate model and run one paired evaluation against your production prompt, then slice the result by whatever case classes you already record. Do not look at the headline number. Look for the class that moved most, and ask whether the instruction responsible was written for a model you still run.