advanced35 minby DevFox

Guardrails and Policy Enforcement

Input and output filters, allowlists, and classifiers — where each belongs in the stack, what they genuinely stop, and why a guardrail that runs on the same model it guards is not a control.

AI
Security

The postmortem ends the way they usually do, with the one action item everybody in the room can agree to: add a guardrail. Three weeks later there are four. One is a regex, one is a small classifier, one is a paragraph appended to the system prompt, and nobody can tell you which of them has ever stopped anything. Latency is up most of a second. Support has a config flag that turns the whole layer off for accounts that complain, and it has been used eleven times this month.

That is the normal end state, and it comes from treating a guardrail as a thing you add rather than as a component with an enforcement point, a threshold, a failure mode and a measured error rate. By the end of this lesson you will be able to say, for every check in your stack: what it runs on, what it costs when it is wrong in each direction, what it does when it breaks, and whether it is a control or a filter. You will also know which of your requirements do not belong in this layer at all.

Four places a check can live

A guardrail is any check outside the model's own generation that can block, alter or escalate a request or a response. The phrase outside the model's own generation is carrying most of the definition, and the rest of the lesson is about why.

There are four kinds, and they differ by where they sit rather than by how clever they are.

Deterministic checks
Ordinary code: a regex, an allowlist, schema validation on a tool argument, a length cap, a rate limit. Identical every time, and it costs microseconds.
It catches only what you were able to write down in advance.
Input classifiers
Learned models over the incoming request. They catch what you can describe but not spell — this message is trying to get the assistant to adopt another persona, this attachment reads like an instruction rather than a document.
Probabilistic, and they run before you spend the expensive call.
Output classifiers
The same machinery over what the model produced, and the only place some questions can be asked at all: whether this reply quotes a record belonging to somebody other than the person asking is not knowable from the request.
Application-layer policy
The check at the site of the action, inside the code holding the credential. Not a filter — a condition on execution.
The refund service compares the amount against the order total whether the caller is a model, a cron job, or a support agent in an admin panel.

Four kinds, differing by where they sit rather than by how clever they are

Think about an airport. The boarding pass scanner is deterministic — the barcode either resolves to a seat on this flight or it does not. Behavioural screening is a classifier: it works on most people, misses some, and stops some travellers who were entirely fine. The reinforced cockpit door is application-layer policy. It does not assess anyone. It just does not open. When a team says "we added guardrails," they almost always mean the middle two. The door is the one that bounds the outcome.

What separates a check from a control

Here is the one rule that survives every change of vendor, model and framework. A guardrail implemented as an instruction to the same model call it guards is not a control.

The reason is mechanical rather than moral. The check and the thing being checked are one forward pass over one token sequence. There is a single input stream, and it governs both outputs. Any text that can steer the answer can steer the verdict on the answer, by construction — and it does not have to be adversarial to do it. A long, badly-ordered context that pushes the reply off course pushes the self-assessment off course with it, in the same direction, for the same reason.

Bad — the verdict and the reply come out of the same call, so whatever moves one moves the other.

Good — the same rule, evaluated by a call the reply cannot write, and acted on by your code.

The bad version produces its own alibi: a reply that breaks the rule arrives with safe: true attached, written by the process that broke it, and the only record of the failure is a field the failure generated. You cannot audit that. You cannot even count it.

It is worth being precise about what the second version buys, because it is easy to oversell. It does not buy an unfoolable guard. The guard reads the same text and can be talked out of the same verdict. What it buys is narrowing: the guard's entire output is one constrained value handed to your code. Whatever an attacker achieves against it is bounded by what that one value can do, which is let a single request through. The guard has no tools, its text is never shown to anyone, and it cannot rewrite the reply. A self-check has no such narrowing, because the channel that carries the verdict is the channel that carries the answer.

One second-order point. Running the guard on the same model family as the generator is common and often correct — shared weights mean correlated blind spots, which is a quality problem you can measure, not a shared channel, which is a structural one you cannot. Where you have the option, a smaller and different model makes a better guard: it is cheaper, it is faster, and its mistakes correlate less with the generator's.

Ordering, and the cost of getting it wrong

Two rules set the order, and they can disagree.

The first is cost. A regex is microseconds, a small classifier is tens of milliseconds, a full model call is seconds. Put the cheap checks first and short-circuit on a definite answer. This is not micro-optimisation: an input check that rejects three percent of traffic before the expensive call removes three percent of your generation spend along with three percent of your risk, and it is the only guardrail in the stack that pays for itself in cash.

The second rule is certainty, and it outranks cost when they conflict. A check whose verdict is definitionally correct must never sit behind a probabilistic one. If a probabilistic check allows the request, you have learned nothing and the definite check still has to run; if it blocks, it has hidden a certain answer behind a guess. Schema validation on a tool argument goes first even when a classifier ahead of it would have been cheaper, because a malformed argument is a fact and a suspicion is not.

The output side has a complication the input side does not. An output classifier wants a complete response; streaming wants to show the first token the moment it exists. There are three honest resolutions and no fourth. Buffer the whole response and pay the latency. Scan incrementally at chunk boundaries and accept that you are ruling on partial text. Or stream and retract — which is worse than it sounds, because the tokens were on the screen, and because retraction does not exist at all when the "stream" is an API response your caller has already parsed.

Choose per surface, not globally. Buffer where the reply can carry the thing the guard exists to catch; stream where it cannot.

A filter that blocks real work gets ignored

Every learned check has a threshold, and moving it trades false negatives for false positives. That trade is a product decision, not a tuning detail, because the two errors are not symmetric in who finds out.

A false negative is silent. Something got through, nobody noticed, and you will learn about it during an incident or never. A false positive is loud and immediate: a customer who was doing their job got refused, today, and there is a person attached to it.

Put a number on it. Forty thousand requests a day through a check with 99.5% precision on legitimate traffic sounds excellent. It is two hundred wrongly blocked requests every day. Within a month somebody has a flag that turns the check off for a named account, or the threshold gets nudged with no measurement behind it, or an exception list appears and starts growing. That is the real failure mode of this layer, and it is not being bypassed. It is being disabled, by your own colleagues, for good reasons.

The way out is to stop treating the decision as one dial with two positions. Give yourself a ladder:

The ladder lets you use a signal whose precision is merely fair at a rung where being wrong is cheap. A 0.7-precision score that quietly removes the refund tool for one turn costs you a slightly worse answer. The identical score wired to a hard block costs you three support tickets an hour and, eventually, its own existence.

So the order of operations is: pick the rung from what a wrong block costs the person on the other end, then pick the threshold from your labelled set at that rung. Never the reverse, and never a number somebody liked the look of.

Fail-open and fail-closed, chosen per action

Your guard is a service. It times out, it returns a 500, its provider has a bad ten minutes. There are exactly two things you can do when a check cannot render a verdict — fail open and let the request continue, or fail closed and refuse it — and choosing once for the whole system is wrong in both directions.

Bad — one try/except in the middleware, so every guard failure everywhere is an allow.

Good — each check declares what its own failure means, at the point it is defined.

The bad version is written by someone who has just watched the product go down because a guard timed out, which makes it a sympathetic mistake rather than a careless one. It still converts every future outage of the guard into an open door on precisely the action you most wanted checked, and the only trace is a warning line in a log nobody reads during an outage.

The rule that falls out: fail closed where the action is irreversible, privileged, or low-volume enough that stopping it is survivable. Fail open where the check is advisory and a halted session costs more than a miss. Then accept the bill honestly — a fail-closed check has made the guard's uptime into your product's uptime for that action.

Which is an argument about which checks should be fail-closed, not just whether. Prefer the fail-closed rung to sit on a deterministic check running in your own process: an amount limit compared against a number you already have has no provider to be down, no timeout, and no bad ten minutes. Let the learned checks, with their network calls and their queues, be the ones that fail open into a flag.

Evaluate a guardrail like the classifier it is

A guardrail is a classifier you have deployed into the critical path of your product, and it deserves everything you would give any other one: a labelled set, an operating point, and a number you re-measure.

It needs its own labelled set, separate from the feature's. Violations are rare in ordinary traffic, so every metric computed over it is dominated by the negative class — a guard that returns "allow" unconditionally will post a superb accuracy figure on your golden set. Report precision and recall at the threshold you actually ship, and report them per policy: recall on jailbreak attempts and recall on third-party data are different numbers with different consequences, and averaging them hides both.

Three sources fill that set. Real traffic labelled by hand, which comes out of your block log. Attempts from your own red-teaming. And the part almost everyone skips: hard negatives, meaning legitimate requests that look like violations. A security researcher genuinely asking how your filters work. A customer quoting the abusive message they received, inside a complaint about receiving it. A clinician asking about a drug interaction. Without hard negatives your precision is measured on traffic that was never going to be blocked, and it will be optimistic in exactly the direction that costs you customers.

A guard drifts for reasons the feature it protects does not. Traffic changes underneath it. An upstream prompt change alters the distribution of text an output guard sees — make replies more verbose and anything keyed on quoted material fires more often, with no change to the guard at all. A model upgrade does the same. So re-measure on a schedule and on every change to what feeds the guard, which includes plenty of changes nobody would describe as touching it.

Log the block, not just the count

A block that a user disputes and you cannot explain is worse than no block at all. Support has nothing to say beyond an apology, and the engineer's only move is to try to reproduce a non-deterministic event from a description. Record enough, at the moment of the decision, that the question is a query instead of an investigation.

The reason code carries more weight than its size suggests. It is the thing that turns a wrong block from an anecdote into a row you can search, group and count — which is how you discover that the same check produced forty of this week's complaints.

Then watch the block rate per check the way you watch an error rate, and alert in both directions. A deploy that moves a check from 0.4% to 9% is an incident even though nothing threw and no page fired. The same number collapsing toward zero is how you find out a check broke, or that an upstream change stopped routing traffic through it at all — a failure that is otherwise completely invisible, because a guard that never fires looks identical to a guard that never needed to.

Guardrails narrow, they never bound

Now the limit, stated plainly, because everything above is only useful if you are honest about this.

Every check in the first three categories is a statement about probability. It moves mass away from bad outcomes. There is no floor beneath it, because it is a pattern or a model reacting to text that an adversary is free to rewrite, and the only evidence that it holds is measurements taken on inputs you have already seen.

There is a test for this, and it takes ten seconds. Write your guarantee as a sentence with no hedge in it. "The assistant will not disclose another customer's record." If the only thing behind that sentence is a classifier at 0.88 recall, the true sentence is "usually does not disclose," and usually is not a guarantee. It is a rate.

So: anything that must not happen belongs in code the model cannot reach. Not a check that inspects the model's output — a condition on the effect, inside the service that performs it, evaluated from data the model did not supply. A refund capped against an order total the payment service reads for itself. Records scoped by a query filtered on the authenticated customer id, so a reply quoting a different customer is not something you catch on the way out but something that was never retrievable on the way in. The strongest control is never the one that refuses well. It is the one where there was nothing available to refuse.

That leaves a fair question: if the layer cannot bound anything, why build it? Because narrowing a distribution is worth real money and real safety, as long as you never call it something else.

It removes the sloppy majority cheaply, so your expensive layers and your humans see far less. It converts a class of behaviour you would otherwise argue about in meetings into a number with a threshold and a trend. It buys time between the day you learn about a new attack class and the day the structural fix ships — a pattern deployed in an afternoon is genuinely valuable while a schema change moves through review. And an output classifier catches the honest failure that no application-layer policy would have seen: the model faithfully summarising an internal-only document that was mistagged as public, where every permission check passed because the data itself was wrong.

None of that is nothing. It is not a bound, and the day you start describing it as one is the day it becomes the only thing standing between an attacker and an action you promised could not happen.

A cheat sheet to keep

Where to go next

You can now describe your enforcement layer as engineering rather than as reassurance: four kinds of check with four enforcement points, an order that follows from cost and certainty, a rung chosen by what a wrong block costs, a failure mode declared per action, a precision number you measured, and a log that makes a wrong block diagnosable. Plus the sentence that outranks all of it — whatever must not happen goes somewhere the model cannot reach.

The natural follow-on is Human-in-the-Loop Design, which picks up the rung this lesson defined but did not staff. Confirm is only a control if the person confirming is actually deciding, and the lesson is about the queue behind it: how escalations are sized, how confidence thresholds route work to people, and the failure where a reviewer approving three hundred items an hour has become a rubber stamp that adds latency and removes nothing.

Before that, do the hour of work that changes how you see your own stack. Pull last week's blocks for your highest-tier check, sample thirty at random, and label each one yourself as violation or legitimate. At the end you have your real precision number and the first thirty rows of a labelled set you did not previously have. Most teams find the number well below what they assumed — and the mislabelled samples tell them, specifically, which rung that check should have been on all along.