intermediate30 minby DevFox

Structured Output You Can Trust

Schema-constrained generation, validating at the boundary, repair loops that converge instead of looping forever, and handling partial objects while a response is still streaming.

Your triage service has been turning support emails into tickets for six weeks and nobody has thought about it once. Then a customer forwards a nine-hundred-word thread, and the reply opens with a sentence of apology before the first brace. The parse throws. The retry throws. The third attempt apologises differently and throws again, and by the time you are awake the queue is nine hundred deep.

The problem is not that the model got one input wrong. It is that nothing in the system had an opinion about what happens next. By the end of this lesson you will be able to make the shape impossible to break, recognise the exact point where that guarantee stops helping you, build a repair loop that ends, and say which fields of a half-arrived object you are allowed to act on.

Three ways a reply can be wrong

You already know to ask for a shape, keep the fields few and flat, and validate what comes back. Validation answers yes or no. The useful thing is that "no" has two very different causes, and "yes" is not the same as "correct".

Malformed
A prose preamble, a markdown fence, a trailing comma, a missing required key, an object cut off at the token limit.
Cheap to detect and, as the next section shows, mostly preventable.
Well-formed and wrong-valued
It parses, it matches the schema, and the value is still unusable. urgency: "high" on a thank-you note. An order_id that is a perfectly good string and refers to no order you ever sold.
A schema is a shape checker and has never had an opinion about meaning. Checking values against the world catches some of it.
Well-formed, well-valued, and false
Plausible values, untrue content. Nothing in this lesson touches it — that one belongs to Grounding and Citation.

Three failures, three different defences — and a schema only helps with the first

Constrained decoding and where it stops

A model produces one token at a time by sampling from a distribution over its whole vocabulary. Constrained decoding — you will also see it called structured generation or schema-constrained generation — compiles your schema into a state machine and, at every step, sets the probability of any token that could not legally continue the document to zero.

Follow it through on the triage object. Before the first token exactly one continuation is legal: {. There is no way to emit an apology, because the letter "I" is not a legal first character. Inside a string the closing brace is unavailable until the quote closes. If a field is an enum of billing, bug and account, then once "b has been emitted the only surviving continuations are illing" and ug" — the model cannot invent a fourth category, because the tokens that would spell one have been masked out of existence. And the closing brace stays illegal until every required key has appeared.

That is a real guarantee, and it deletes most of the first failure class outright. It has two edges worth knowing precisely.

It does not guarantee you got a whole document. Constrained decoding guarantees that every prefix can be extended into a valid document, not that the extension happened. Hit your max-token limit mid-object and you get a valid prefix with no closing brace, which fails to parse exactly like the messy old failures did.

And the mask is applied to the sampler, not to the model's beliefs. Where the model wanted a token the schema forbids, it gets its second choice instead, and every token after that is conditioned on a token it did not want. Usually that is invisible. Sometimes it is the whole problem, which is the next section.

When the shape fights the task

A schema is an instruction the model cannot argue with. That is the point, and it is also the failure.

Consider a required order_id typed as a string, on an email that contains no order number. The model has one legal move: emit a string that looks like an order number. It cannot say "there isn't one" — those tokens are masked. It cannot leave the field out — the closing brace is masked until the key appears. So it invents ORD-4417, the object validates, and the field lands in your database indistinguishable from forty thousand real ones.

Notice what you actually did. Before the constraint, the model wrote a paragraph saying it could not find an order number, and that paragraph was a signal. You did not remove the guess. You removed the tell.

The second version of this problem is space to think. If the constraint starts at the first token, there is nowhere to work anything out before committing. Fields are generated in schema order, so put a short free-text field first — evidence, holding the sentence that justifies the classification — and let the model fill it before it commits to the enum. Or run an unconstrained pass into a constrained extraction pass, which costs a call and buys a model allowed to change its mind.

Give the model somewhere to say no

The rule that prevents most of this is one question asked of every field: what should this be when the input does not contain it? If you cannot answer, the field is not required.

Bad — an email with no order number still comes back with an order number.

Good — absence is a value the schema can express, so the model has a legal way to report it.

A None tells your agent to ask the customer for their order number. ORD-4417 sends that agent to look up an order nobody ever placed, and the first schema gave you no way to tell those two cases apart.

Nullability handles a missing field. A missing answer wants something stronger: a place for the model to say the task itself could not be done. A discriminated union does this without filling a real object with nulls.

Now "this email is an out-of-office autoreply" has a shape of its own, and your code branches on status instead of guessing from a pile of nulls whether it got a real classification. Constrained decoding handles unions fine: once status is chosen, the state machine permits only that branch's fields.

A repair loop that converges

Some replies still fail validation: your runtime may not offer constrained decoding for the model you need, or the failure is semantic and only your own checks catch it. So you retry.

Retry here means one thing only: the output violated the schema. A timeout, a rate limit, a filtered response or a provider outage is a different failure with a different policy, and Failure Modes, Retries, and Fallbacks is where those live. Keep them apart. One counter covering both is a number that means nothing and a budget you cannot reason about.

A repair loop has to change something on each pass. Resending the identical prompt is a slot machine — same context, same temperature, different sample. It sometimes works, which is the worst outcome available, because now you believe retrying works.

Bad — five identical calls, no new information, and the caller waits for every one of them.

Good — each attempt carries back the exact thing the validator objected to, and the loop is bounded.

A malformed reply produced from an unchanged prompt fails the same way five times, so the first version turns one bad answer into five times the latency and five times the cost and still hands the caller nothing.

Pass the validator's error verbatim, not a summary of it. A message naming the path, the expected type and what arrived — urgency: input should be 'low', 'normal' or 'high', got 'urgent' — is a mechanical correction the model can apply. "That was wrong, try again" is another roll of the dice. Truncate it if you must; an error over a large object can outrun the object.

Bound the loop at one repair attempt for constrained output and two for unconstrained. The reason is not thrift. If the second attempt fails the way the first did, the cause is your schema or your input, and neither changes on the third call. A climbing repair rate is the best signal you get that the schema has drifted away from the task it describes.

The fallback you chose before you shipped

The loop above has exactly two exits: a validated object, or None. The second is not an accident to handle later — it is a product decision, and if you cannot say in one sentence what the caller gets when repair fails, the feature is not finished.

A good fallback is deterministic, cheap, and does not call a model. That last part matters more than it sounds: a fallback that calls a model is a second thing that can fail, at the same moment, often for the same reason, and now your failure path has a failure path.

For the triage service it is one line of code and a sentence of policy: route the email to the human queue as unclassified, raw text attached. That is worse than a good extraction and enormously better than a confident wrong one. A degraded deterministic path is fair game too — keyword rules that set category and leave urgency null — provided the result is labelled as coming from the rules.

Write the test that forces the loop to exhaust. Stub the model to return garbage, run it, assert the ticket landed in the human queue. Without it, the first real execution of your fallback path is in production, at the worst moment, with nobody watching.

Streaming: what is safe to act on

You stream a structured reply for one reason: the user watches the summary form instead of a spinner. The cost is that JSON is only valid at the very end. Between the braces you hold a prefix, and a prefix is not a document — you cannot run it through a schema with required fields, because those fields have not been said yet.

So you parse leniently. For a flat object that is a few lines: close the structure and see whether what you have parses.

Now the subtle part: which of those values are settled. A strict parser like the one above never hands you half a string, because an unclosed quote does not parse — but it will cheerfully hand you 12 while the model is halfway through writing 125, since a truncated number is still a number. A lenient library parser, the kind built for rendering text as it arrives, closes the open quote so you can display a half-written summary; that convenience is exactly what makes its newest value unsafe to compare against anything, because "billing-escalations" passes through the state "billing" on its way to being itself, and "billing" is a legal queue.

Either parser, the rule is the same. The newest pair in the buffer is still being written. Drop it.

Bad — assigns the ticket from a value read out of an object that has not finished existing.

Good — shows the partial object to the user, and acts only on one that parsed and validated.

In the first version the object may never validate at all — a later required field is missing, the repair loop runs, and the corrected object names a different queue. The ticket was assigned four hundred milliseconds ago, and nothing in the correct final object undoes it.

The line to hold is that rendering is reversible and side effects are not. Redrawing a summary that changed under the user costs nothing; issuing a refund on a value that was still being typed costs the refund, and you find out afterwards.

Two habits help. Put long free-text fields last in the schema, so the short machine-actionable ones settle within the first few hundred milliseconds. And send the finished buffer through the same validator your non-streaming path uses, so there is one place where an object becomes trustworthy rather than two implementations quietly disagreeing.

A cheat sheet to keep

Where to go next

You can now put a hard floor under the shape of a reply, keep the schema from forcing an invention, repair only what is worth repairing and only as often as that is worth doing, and tell a value you may display from a value you may act on.

The next lesson to read is Failure Modes, Retries, and Fallbacks, which takes the retries this one deliberately refused: timeouts, rate limits, content filters, provider outages, and output truncated at the token limit. Those need backoff, sometimes a different model, and occasionally an immediate error to the user — a policy that looks nothing like feeding a validation message back into the same conversation.

The thing to go and do is small and will surprise you. Take an extractor you already run, make one required field nullable, and add a short field where the model can name what it could not find in the input. Re-run it over fifty real records and count how often the escape hatch gets used. Every one of those was a fabricated value that passed validation yesterday.