Guardrails and Policy Enforcement
Input and output filters, allowlists, and classifiers — where each belongs in the stack, what they genuinely stop, and why a guardrail that runs on the same model it guards is not a control.
Input and output filters, allowlists, and classifiers — where each belongs in the stack, what they genuinely stop, and why a guardrail that runs on the same model it guards is not a control.
The postmortem ends the way they usually do, with the one action item everybody in the room can agree to: add a guardrail. Three weeks later there are four. One is a regex, one is a small classifier, one is a paragraph appended to the system prompt, and nobody can tell you which of them has ever stopped anything. Latency is up most of a second. Support has a config flag that turns the whole layer off for accounts that complain, and it has been used eleven times this month.
That is the normal end state, and it comes from treating a guardrail as a thing you add rather than as a component with an enforcement point, a threshold, a failure mode and a measured error rate. By the end of this lesson you will be able to say, for every check in your stack: what it runs on, what it costs when it is wrong in each direction, what it does when it breaks, and whether it is a control or a filter. You will also know which of your requirements do not belong in this layer at all.
A guardrail is any check outside the model's own generation that can block, alter or escalate a request or a response. The phrase outside the model's own generation is carrying most of the definition, and the rest of the lesson is about why.
There are four kinds, and they differ by where they sit rather than by how clever they are.
Deterministic checks
Ordinary code: a regex, an allowlist, schema validation on a tool argument, a length cap, a rate limit. Identical every time, and it costs microseconds.
It catches only what you were able to write down in advance.
Input classifiers
Learned models over the incoming request. They catch what you can describe but not spell — this message is trying to get the assistant to adopt another persona, this attachment reads like an instruction rather than a document.
Probabilistic, and they run before you spend the expensive call.
Output classifiers
The same machinery over what the model produced, and the only place some questions can be asked at all: whether this reply quotes a record belonging to somebody other than the person asking is not knowable from the request.
Application-layer policy
The check at the site of the action, inside the code holding the credential. Not a filter — a condition on execution.
The refund service compares the amount against the order total whether the caller is a model, a cron job, or a support agent in an admin panel.
Think about an airport. The boarding pass scanner is deterministic — the barcode either resolves to a seat on this flight or it does not. Behavioural screening is a classifier: it works on most people, misses some, and stops some travellers who were entirely fine. The reinforced cockpit door is application-layer policy. It does not assess anyone. It just does not open. When a team says "we added guardrails," they almost always mean the middle two. The door is the one that bounds the outcome.
Here is the one rule that survives every change of vendor, model and framework. A guardrail implemented as an instruction to the same model call it guards is not a control.
The reason is mechanical rather than moral. The check and the thing being checked are one forward pass over one token sequence. There is a single input stream, and it governs both outputs. Any text that can steer the answer can steer the verdict on the answer, by construction — and it does not have to be adversarial to do it. A long, badly-ordered context that pushes the reply off course pushes the self-assessment off course with it, in the same direction, for the same reason.
Bad — the verdict and the reply come out of the same call, so whatever moves one moves the other.
SYSTEM = """You are a support assistant for order issues.
Before replying, check your own answer: if it names any
customer other than the one asking, do not send it.
Respond as JSON: {"safe": bool, "reply": str}"""
def answer(ticket: Ticket) -> str:
result = model.json(SYSTEM, ticket.thread)
return result["reply"] if result["safe"] else FALLBACKGood — the same rule, evaluated by a call the reply cannot write, and acted on by your code.
def answer(ticket: Ticket) -> str:
reply = model.text(ASSISTANT_SYSTEM, ticket.thread)
verdict = guard.classify( # own prompt, own call
policy="names_third_party",
content=reply,
)
if verdict.score >= NAMES_THIRD_PARTY_BLOCK:
record_block("names_third_party", verdict, ticket)
return FALLBACK
return replyThe bad version produces its own alibi: a reply that breaks the
rule arrives with safe: true attached, written by the process
that broke it, and the only record of the failure is a field the
failure generated. You cannot audit that. You cannot even count
it.
It is worth being precise about what the second version buys, because it is easy to oversell. It does not buy an unfoolable guard. The guard reads the same text and can be talked out of the same verdict. What it buys is narrowing: the guard's entire output is one constrained value handed to your code. Whatever an attacker achieves against it is bounded by what that one value can do, which is let a single request through. The guard has no tools, its text is never shown to anyone, and it cannot rewrite the reply. A self-check has no such narrowing, because the channel that carries the verdict is the channel that carries the answer.
One second-order point. Running the guard on the same model family as the generator is common and often correct — shared weights mean correlated blind spots, which is a quality problem you can measure, not a shared channel, which is a structural one you cannot. Where you have the option, a smaller and different model makes a better guard: it is cheaper, it is faster, and its mistakes correlate less with the generator's.
Two rules set the order, and they can disagree.
The first is cost. A regex is microseconds, a small classifier is tens of milliseconds, a full model call is seconds. Put the cheap checks first and short-circuit on a definite answer. This is not micro-optimisation: an input check that rejects three percent of traffic before the expensive call removes three percent of your generation spend along with three percent of your risk, and it is the only guardrail in the stack that pays for itself in cash.
The second rule is certainty, and it outranks cost when they conflict. A check whose verdict is definitionally correct must never sit behind a probabilistic one. If a probabilistic check allows the request, you have learned nothing and the definite check still has to run; if it blocks, it has hidden a certain answer behind a guess. Schema validation on a tool argument goes first even when a classifier ahead of it would have been cheaper, because a malformed argument is a fact and a suspicion is not.
INPUT_CHECKS = [
RequestSizeLimit(max_tokens=8000), # µs, exact
AttachmentTypeAllowlist(ALLOWED_MIME), # µs, exact
InjectionClassifier(threshold=0.92), # ~30ms, learned
]The output side has a complication the input side does not. An output classifier wants a complete response; streaming wants to show the first token the moment it exists. There are three honest resolutions and no fourth. Buffer the whole response and pay the latency. Scan incrementally at chunk boundaries and accept that you are ruling on partial text. Or stream and retract — which is worse than it sounds, because the tokens were on the screen, and because retraction does not exist at all when the "stream" is an API response your caller has already parsed.
Choose per surface, not globally. Buffer where the reply can carry the thing the guard exists to catch; stream where it cannot.
Every learned check has a threshold, and moving it trades false negatives for false positives. That trade is a product decision, not a tuning detail, because the two errors are not symmetric in who finds out.
A false negative is silent. Something got through, nobody noticed, and you will learn about it during an incident or never. A false positive is loud and immediate: a customer who was doing their job got refused, today, and there is a person attached to it.
Put a number on it. Forty thousand requests a day through a check with 99.5% precision on legitimate traffic sounds excellent. It is two hundred wrongly blocked requests every day. Within a month somebody has a flag that turns the check off for a named account, or the threshold gets nudged with no measurement behind it, or an exception list appears and starts growing. That is the real failure mode of this layer, and it is not being bypassed. It is being disabled, by your own colleagues, for good reasons.
The way out is to stop treating the decision as one dial with two positions. Give yourself a ladder:
allow score below the flag threshold
allow + log flagged; nothing changes for the user
degrade answer, but drop the write tools this turn
confirm answer, require a human OK before the effect
block refuse, with a reason code the user can quoteThe ladder lets you use a signal whose precision is merely fair at a rung where being wrong is cheap. A 0.7-precision score that quietly removes the refund tool for one turn costs you a slightly worse answer. The identical score wired to a hard block costs you three support tickets an hour and, eventually, its own existence.
So the order of operations is: pick the rung from what a wrong block costs the person on the other end, then pick the threshold from your labelled set at that rung. Never the reverse, and never a number somebody liked the look of.
Your guard is a service. It times out, it returns a 500, its provider has a bad ten minutes. There are exactly two things you can do when a check cannot render a verdict — fail open and let the request continue, or fail closed and refuse it — and choosing once for the whole system is wrong in both directions.
Bad — one try/except in the middleware, so every guard failure everywhere is an allow.
def run_checks(request: Request) -> Decision:
try:
return guard.evaluate(request, timeout=2.0)
except Exception:
logger.warning("guard unavailable, allowing")
return Decision.allow()Good — each check declares what its own failure means, at the point it is defined.
@dataclass(frozen=True)
class Check:
name: str
evaluate: Callable[[Request], Decision]
on_error: Decision # declared, never defaulted
CHECKS = [
Check("pii_in_reply", scan_pii,
on_error=Decision.allow_and_flag()),
Check("refund_within_policy", scan_refund,
on_error=Decision.block("guard_unavailable")),
]The bad version is written by someone who has just watched the product go down because a guard timed out, which makes it a sympathetic mistake rather than a careless one. It still converts every future outage of the guard into an open door on precisely the action you most wanted checked, and the only trace is a warning line in a log nobody reads during an outage.
The rule that falls out: fail closed where the action is irreversible, privileged, or low-volume enough that stopping it is survivable. Fail open where the check is advisory and a halted session costs more than a miss. Then accept the bill honestly — a fail-closed check has made the guard's uptime into your product's uptime for that action.
Which is an argument about which checks should be fail-closed, not just whether. Prefer the fail-closed rung to sit on a deterministic check running in your own process: an amount limit compared against a number you already have has no provider to be down, no timeout, and no bad ten minutes. Let the learned checks, with their network calls and their queues, be the ones that fail open into a flag.
A guardrail is a classifier you have deployed into the critical path of your product, and it deserves everything you would give any other one: a labelled set, an operating point, and a number you re-measure.
It needs its own labelled set, separate from the feature's. Violations are rare in ordinary traffic, so every metric computed over it is dominated by the negative class — a guard that returns "allow" unconditionally will post a superb accuracy figure on your golden set. Report precision and recall at the threshold you actually ship, and report them per policy: recall on jailbreak attempts and recall on third-party data are different numbers with different consequences, and averaging them hides both.
guard: names_third_party, threshold 0.90, n = 1,200
blocked allowed
actual violation 44 (TP) 6 (FN)
legitimate 31 (FP) 1,119 (TN)
accuracy 0.969 <- useless; allow-all scores 0.958
precision 0.587 <- 4 of every 10 blocks was a customer
recall 0.880Three sources fill that set. Real traffic labelled by hand, which comes out of your block log. Attempts from your own red-teaming. And the part almost everyone skips: hard negatives, meaning legitimate requests that look like violations. A security researcher genuinely asking how your filters work. A customer quoting the abusive message they received, inside a complaint about receiving it. A clinician asking about a drug interaction. Without hard negatives your precision is measured on traffic that was never going to be blocked, and it will be optimistic in exactly the direction that costs you customers.
A guard drifts for reasons the feature it protects does not. Traffic changes underneath it. An upstream prompt change alters the distribution of text an output guard sees — make replies more verbose and anything keyed on quoted material fires more often, with no change to the guard at all. A model upgrade does the same. So re-measure on a schedule and on every change to what feeds the guard, which includes plenty of changes nobody would describe as touching it.
A block that a user disputes and you cannot explain is worse than no block at all. Support has nothing to say beyond an apology, and the engineer's only move is to try to reproduce a non-deterministic event from a description. Record enough, at the moment of the decision, that the question is a query instead of an investigation.
request_id ties the block to the request's full trace
check which check fired, by stable name
policy_version threshold plus rule or guard model version
score the raw value and the threshold it crossed
evidence rule id, matched offsets, or the label
action allow / flag / degrade / confirm / block
user_message the exact text the person was shown
reason_code short, stable, quotable in a support ticketThe reason code carries more weight than its size suggests. It is the thing that turns a wrong block from an anecdote into a row you can search, group and count — which is how you discover that the same check produced forty of this week's complaints.
Then watch the block rate per check the way you watch an error rate, and alert in both directions. A deploy that moves a check from 0.4% to 9% is an incident even though nothing threw and no page fired. The same number collapsing toward zero is how you find out a check broke, or that an upstream change stopped routing traffic through it at all — a failure that is otherwise completely invisible, because a guard that never fires looks identical to a guard that never needed to.
Now the limit, stated plainly, because everything above is only useful if you are honest about this.
Every check in the first three categories is a statement about probability. It moves mass away from bad outcomes. There is no floor beneath it, because it is a pattern or a model reacting to text that an adversary is free to rewrite, and the only evidence that it holds is measurements taken on inputs you have already seen.
There is a test for this, and it takes ten seconds. Write your guarantee as a sentence with no hedge in it. "The assistant will not disclose another customer's record." If the only thing behind that sentence is a classifier at 0.88 recall, the true sentence is "usually does not disclose," and usually is not a guarantee. It is a rate.
So: anything that must not happen belongs in code the model cannot reach. Not a check that inspects the model's output — a condition on the effect, inside the service that performs it, evaluated from data the model did not supply. A refund capped against an order total the payment service reads for itself. Records scoped by a query filtered on the authenticated customer id, so a reply quoting a different customer is not something you catch on the way out but something that was never retrievable on the way in. The strongest control is never the one that refuses well. It is the one where there was nothing available to refuse.
That leaves a fair question: if the layer cannot bound anything, why build it? Because narrowing a distribution is worth real money and real safety, as long as you never call it something else.
It removes the sloppy majority cheaply, so your expensive layers and your humans see far less. It converts a class of behaviour you would otherwise argue about in meetings into a number with a threshold and a trend. It buys time between the day you learn about a new attack class and the day the structural fix ships — a pattern deployed in an afternoon is genuinely valuable while a schema change moves through review. And an output classifier catches the honest failure that no application-layer policy would have seen: the model faithfully summarising an internal-only document that was mistagged as public, where every permission check passed because the data itself was wrong.
None of that is nothing. It is not a bound, and the day you start describing it as one is the day it becomes the only thing standing between an attacker and an action you promised could not happen.
FOUR KINDS, BY ENFORCEMENT POINT
deterministic regex, allowlist, schema, limits. exact,
microseconds, catches only what you wrote down
input class. learned, before the expensive call. cuts
spend as well as risk
output class. learned, over the reply. the only place a
leak in *this* answer is visible at all
app policy a condition inside the code holding the
credential. the only kind that bounds
THE RULE
a check the guarded call writes itself is not a control:
one input stream governs the answer and the verdict
a separate call is foolable too, but its whole output is
one label your code consumes, so the damage is one request
prompt hardening = mitigation. keep it, stop counting it
ORDER
cheap before expensive microseconds -> ms -> seconds
certain before probabilistic never hide a definite verdict
streaming out: buffer, chunk-scan, or retract. per surface
give the guard layer its own timeout and its own p99 line
THE THRESHOLD IS A PRODUCT DECISION
false negative silent, found in an incident, or never
false positive today, on a customer, with a person attached
the ladder allow / flag / degrade / confirm / block
pick the rung from what a wrong block costs, then pick the
threshold from the labelled set at that rung
a filter that blocks legitimate work gets switched off
per-account bypasses are how a guardrail dies
FAILURE MODE, PER CHECK, WRITTEN DOWN
fail closed irreversible, privileged, low volume
fail open advisory, high volume, cheap to be wrong
fail-closed checks should have the fewest dependencies
fail closed = the guard's uptime is now the product's
EVALUATION
its own labelled set, not the feature's golden set
precision and recall per policy, at the shipped threshold
accuracy is meaningless: allow-everything scores ~0.96
hard negatives are the ones you are missing
re-measure on upstream prompt and model changes too
LOGGING
request_id, check, policy_version, score, evidence,
action, user_message, reason_code
block rate per check, alerted in both directions
the block log holds what the guard exists to keep out:
shortest retention, tightest access, no shared sink
THE LIMIT
write the guarantee with no hedge in it. if you cannot,
it is a rate, not a control. move it into code the model
cannot reach.You can now describe your enforcement layer as engineering rather than as reassurance: four kinds of check with four enforcement points, an order that follows from cost and certainty, a rung chosen by what a wrong block costs, a failure mode declared per action, a precision number you measured, and a log that makes a wrong block diagnosable. Plus the sentence that outranks all of it — whatever must not happen goes somewhere the model cannot reach.
The natural follow-on is Human-in-the-Loop Design, which picks up the rung this lesson defined but did not staff. Confirm is only a control if the person confirming is actually deciding, and the lesson is about the queue behind it: how escalations are sized, how confidence thresholds route work to people, and the failure where a reviewer approving three hundred items an hour has become a rubber stamp that adds latency and removes nothing.
Before that, do the hour of work that changes how you see your own stack. Pull last week's blocks for your highest-tier check, sample thirty at random, and label each one yourself as violation or legitimate. At the end you have your real precision number and the first thirty rows of a labelled set you did not previously have. Most teams find the number well below what they assumed — and the mislabelled samples tell them, specifically, which rung that check should have been on all along.