intermediate30 minby DevFox

Shipping an AI Feature Safely

Rolling out something you cannot fully predict: flags, canaries, a kill switch, a feedback path from users back into your evaluation set, and what to do the first time it embarrasses you.

AI
DevOps

The eval suite has been green for a week and it reads 91%. Someone asks whether that is good enough to ship and the room goes quiet, because nobody can say what number would be. The missing 9% are not bugs waiting to be closed — some are questions with no clean answer, and the score will never read 100. The branch is merged and everyone is waiting on a decision no dashboard is going to make.

This lesson is about the release rather than the model. By the end you will have a launch bar you can defend out loud, a rollout that fails in front of ten people instead of ten thousand, a kill switch that leaves users something useful, a prompt you can roll back without shipping code, and a route carrying a user's complaint back into the suite that told you it was ready.

Deciding ready when the score never reads 100

A deterministic feature ships when the tests pass, because the tests enumerate the behaviour. Your output space cannot be enumerated — that is why it needed a model — so no score means "correct". It means "correct on the cases we thought of", and the honest question is not is it good enough but what does being wrong cost us, and how often can we afford it.

That question has three terms: how often the feature is wrong, how bad a wrong answer is, and how easily one is caught and undone. The first comes from your suite; the other two are product judgements, yours to make before the run rather than after it.

So stop treating failures as one population. Sort every eval case into a consequence tier. Cosmetic failures — wrong tone, too long — cost you a shrug. Recoverable failures are wrong in a way the user sees at once and can redo. Unrecoverable failures are acted on before anyone checks: an email sent, a record deleted, a refund approved. Those are three different products failing, and one average across them tells you nothing.

Bad — one aggregate bar, so enough easy passes can cover any hard failure.

python

PASS_RATE_REQUIRED = 0.95

def ready_to_ship(results):
    passed = sum(1 for result in results if result.passed)
    return passed / len(results) >= PASS_RATE_REQUIRED

Good — a separate bar per consequence tier, each one able to block on its own.

python

LAUNCH_BARS = {
    "cosmetic": 0.80,       # wrong tone or length; user shrugs
    "recoverable": 0.95,    # visibly wrong, cheap to redo
    "unrecoverable": 1.00,  # acted on before anyone checks
}

def ready_to_ship(results):
    for tier, bar in LAUNCH_BARS.items():
        cases = [r for r in results if r.consequence_tier == tier]
        if sum(r.passed for r in cases) / len(cases) < bar:
            return False
    return True

Ninety-five percent across a mixed bag can hide every unrecoverable case failing, as long as enough tone checks pass to carry the average — and those are the failures you hear about from a customer or a lawyer rather than from a dashboard.

Ship behind a flag, ramp in three stages

Two things get confused into the word "launch". Deploy means the code is running on your servers. Release means a user can reach it. A feature flag keeps them separate — a runtime check deciding, per request, whether this user gets the new path — and three of its properties matter more than which flag system you pick. The check happens per request, not at boot, because a flag read from an environment variable at startup needs a rolling restart to change and a restart is the ten minutes you will not have. It is sticky per user, so nobody gets the new product on one page and the old one on the next, which also keeps your metrics readable. And flipping it takes no deploy.

Once you can decide per user, you can ramp. Each stage answers a question the previous one could not, and knowing which question you are asking stops you reading a stage for evidence it cannot give.

Internal — your team, every request
Does this work at all against real data? This is where you find that the tool never fires in production because a scope is missing, or that the prompt names a field the API stopped sending.
Do not read quality here. Your colleagues ask the questions the demo was built from, and forgive what a stranger will not.
One to five percent, against a holdout
Do the signals hold on inputs nobody imagined? Refusal rate, tool error rate, retry rate, p95 latency, cost per request, and the rate at which users flag an answer.
Read every one as a difference against the holdout. Support volume rises in the first week of the month whatever you ship, and without a control you cannot tell a bad feature from a bad Monday.
Everyone
Does it hold at volume and at cost? Failures here are load-shaped rather than quality-shaped — rate limits you never touched at one percent, a cache hit rate that collapses as the long tail arrives, cost per request drifting up because real inputs run longer than sampled ones.

Each stage answers a question the previous one could not. Reading a stage for evidence it cannot give is the mistake.

How long to sit at each stage is a question about traffic, not headcount. A stage that ran Tuesday to Thursday told you about Tuesday to Thursday; dwell for a full cycle of your own usage.

A kill switch that degrades instead of dying

A breaker that cuts the lights in a hospital is not a safety device. The one you want trips to emergency lighting — less than before, still enough to walk by. A kill switch has the same job: not to stop the feature, but to leave the user something that works.

Decide the degraded state before launch, and pick it from what you already have: the deterministic path that existed before the model arrived — keyword search, a template, a rules engine — or the human queue, or hiding the feature when there is genuinely nothing underneath, which is honest and quiet. What it must never be is a stack trace with your product's name on it.

Bad — turning the feature off takes the page down with it.

Good — off means the product you had before, which was fine.

The first version turns "our summariser is embarrassing us" into "the ticket page is broken", so the cheapest possible response — switching it off — becomes an outage of its own, and the person on call hesitates over the one action that would have ended it.

Then flip it. During the internal stage, turn the switch off in production, look at what a real user sees, and turn it back on. An untested kill switch is a hypothesis, tested for the first time on the worst day of the quarter.

The prompt is a deployable artifact

Your system prompt, tool descriptions, few-shot examples, output schema and retrieval settings are not configuration. They determine the feature's behaviour, change more often than the code around them, and are the usual cause when quality moves. Treat the set as one prompt artifact and give it what you give code: a version in source control, a review before it changes, an identifier, and a rollback. The identifier makes the rest work — every request records which version produced it, so a quality drop becomes a question you can answer rather than a debate.

Bad — the prompt is edited in a console and fetched fresh each request; no trace can say what ran.

Good — a pinned version from the repository, recorded on the trace.

When answers get worse on Tuesday, the first version cannot tell you whether the prompt moved, because "latest" is not a value you can compare against anything — and the person who tightened one sentence at 4pm on Friday has genuinely forgotten they did.

Rolling the prompt back independently of code is the point of all this. Most bad releases of an AI feature are prompt releases, and if the prompt only ships with a deploy, undoing it undoes everything that went out alongside it — including the fix somebody is waiting on. What you owe the rollback is a compatibility check: v7 may expect a field the current code stopped sending, so run the suite against the pair rather than assuming an old prompt is safe.

The feedback path that closes the loop

Here is the difference between a feature that improves and one that merely ages. The world moves — documentation changes, users learn new phrasings, the questions asked in month six are not the ones you sampled in month one. A launch-day eval set measures a launch-day product forever unless something keeps feeding it.

That something is your users, and the path has two halves. The first is the control: one click, where the bad answer appeared, no form and no required explanation. Friction here costs you the reports from people too busy to file them, who are the users you most want to hear from.

The second half is the one teams skip. A flag is only useful if it carries enough to reproduce the failure, so the click attaches to the trace rather than replacing it — trace id, prompt version, retrieved documents, tool calls, the exact output. Then a human triages three questions: was it actually wrong, what should it have said, and is this one case or a class. The answer becomes a new eval case in the right consequence tier, with the correct output written down, and the fix ships when that case passes and none of the old ones broke.

Without that second half, the thumbs-down button is a satisfaction metric. With it, every failure a user finds is a failure that can only happen once.

Say what it is, in the interface

The same wrong sentence produces two entirely different reactions depending on framing. In an editable box with the cursor already in it, a flawed summary is a starting point and the user fixes it without thinking. In a settled panel with a tick beside it, that sentence is a broken product and a support ticket. Nothing about the model changed. A feature presented as certain is judged as certain.

Three levers, all cheap. Name the output — Draft, Suggested, Generated — so the reader knows what kind of thing they have. Make editing the primary action rather than accepting, so the expected interaction is review rather than trust. And match your language to the real reliability: confident phrasing where the suite says you have earned it, visible uncertainty where it does not.

Notice what this does to the first section: it moves the third term in the cost of being wrong. Design that makes errors visible and cheap to fix genuinely lowers the bar the feature has to clear — the only lever here that makes you more able to ship rather than more careful about it.

The first time it embarrasses you

It will happen, and it will not be a case from your suite. Someone screenshots your feature saying something confidently wrong, and the screenshot reaches your CEO before your alerting does. The work of that first hour is a sequence, and the order matters.

Degrade before you diagnose. Flip the flag for the affected cohort, or for everyone. Because the off path is the product you had last month, this costs a feature rather than an outage, and it buys back the time to think.

Capture before it ages. Pull the trace, the prompt version, the retrieved context and the exact output somewhere permanent. Log retention is measured in days, and you will want this in three weeks when someone asks whether the fix worked.

Turn the screenshot into a test. The failing input, with the answer it should have given, goes into the eval set at the right consequence tier before anyone writes a fix — so the suite proves the fix rather than the author's confidence, and this exact embarrassment becomes impossible to reship.

Re-ramp, do not restore. The fix is a change like any other, so walk it back through the stages with a holdout. Jumping straight to everyone because you are sure is how the second incident starts.

Say what happened, plainly. Users already know these systems are wrong sometimes; what they punish is a company that appears not to have noticed.

That is the first hour. The discipline that follows — replaying a request against a different version, reasoning about a root cause that is a probability rather than a line of code, writing the postmortem — is a subject in its own right, waiting for you in the advanced course.

A launch checklist to keep

Where to go next

That is the end of the course. You can shape a context window, give a model tools it uses well, ground its answers, run an agent loop that terminates, defend a trust boundary, evaluate the result and watch it in production — and now release it to real people without betting the product on being right first time.

What comes next is scale and adversaries, and that is Production AI Engineering. It settles the questions this course could only reach: an eval harness whose numbers survive statistical scrutiny, and a model-as-judge that correlates with human labels instead of flattering itself; when splitting work across multiple agents earns its cost and when it is expensive theatre; the full adversarial threat model, where an attacker chains injections through your tools and exfiltrates through their arguments; caching and routing across a stack of models rather than within one call; what to do when a better model breaks your prompts, and how to drive that upgrade as a measured migration; responding to incidents in a system with no stack trace; and the unit economics that quietly decide your architecture whether or not you take part.

Before any of that, do one thing today. Take the AI feature you have running now and try to turn it off — properly, in production, for a real cohort — timing the gap between deciding and the last user seeing the degraded path. If that takes a deploy, or if what they land on is an error, you have found the most valuable thing you could build this week.