intermediate30 minby DevFox

Failure Modes, Retries, and Fallbacks

Timeouts, rate limits, content filters, truncated output and provider outages — which are worth retrying, which need a different model, and which must surface to the user immediately.

Your ticket summariser has run quietly for three months. Then on a Tuesday afternoon the provider starts answering in eleven seconds instead of two, one call in twenty comes back 529, and your queue workers — each patiently retrying five times, one second apart — stop draining. Nothing is down. Support sees no summaries at all, and the backlog grows for forty minutes before anyone works out why.

A model call is a network call to a busy, rate-limited, occasionally degraded service, plus a few failure modes ordinary services do not have: a response that declines to answer, and a response that stops mid-sentence and still arrives with a 200. By the end of this lesson you will name the failure in front of you, answer the one question that decides whether retrying it can help at all, and choose the fallback on purpose rather than inheriting whatever your exception handler happened to do.

Six ways a model call goes wrong

Slow responses are not errors, which is what makes them dangerous. The call succeeds; it takes eleven seconds instead of two. Nothing raises, no alert fires on error rate, and every worker waiting on a socket is a worker doing nothing else. Capacity disappears long before correctness does.

Timeouts are the same event after your client gives up, and they are ambiguous in a way that matters: the request may have completed on the provider's side, generated every token, and been billed to you — you are the only party without the answer. A retry after a timeout is a second full generation, not a resumption of the first.

Rate limits arrive as 429: you asked for more than your allowance, usually measured in both requests and tokens per minute. That allowance is typically shared across a whole account, so another team's overnight batch job can rate-limit your interactive feature at nine in the morning. The response often carries a retry-after header, and that number beats any you would compute.

Transient server errors — 500, 502, 503, and the overloaded-style codes providers use when shedding load — are the provider's problem and usually brief. The textbook retryable case.

Content filter refusals are the first failure with no equivalent in an ordinary API. The request succeeds, the response is well-formed, and the body declines the work: a safety layer or the model itself has ruled this input out of bounds. A customer pasting a threat into a support ticket can trigger one on a summariser that never saw a refusal in testing.

Truncation is the second. The model hit the maximum output tokens you allowed and stopped where it was — mid-sentence, mid-object, mid-list. Status 200. Body present. Content a fragment. The only thing that tells you is the stop reason, the field your provider calls stop_reason or finish_reason, which says whether the model finished its thought or ran out of room.

There is also a seventh thing, a weather condition rather than a failure mode: the provider having a bad hour, where the first four arrive together at elevated rates and nothing is ever cleanly down. It gets its own section, because the right response to it is the opposite of the right response to a blip.

The only question that matters

For every one of those, ask: will doing this again produce a different result? That single question splits the whole list.

A property of the moment

Retry it.

A connection reset, a provider shedding load, a rate limit that expires in four seconds.

The moment passes and the same request gets a real answer. Not retrying turns a two-second blip into a visible outage.

A property of the request

Do not.

A malformed body, a revoked key, an input longer than the context window, a refusal on content that will be exactly as objectionable the second time.

Sending it again reproduces the failure, more slowly and at more cost.

Both directions cost you, and the mistake is usually made in whatever except clause happens to catch first

Make the call explicitly, in one place.

Bad — retries a malformed request four times, so a permanent bug takes four seconds to report and costs four calls.

Good — retries only the failures a retry could change, and re-raises the rest at once.

The blanket version hides the failures you most need to see. A 400 from a prompt that outgrew the context window is a bug you could fix today, and the first version buries it under three pointless retries and reports it as the same unavailability as a network blip.

Timeouts are a product decision

The timeout on a model call is not a technical detail inherited from your HTTP client. It is a statement about how long your user will wait, and it belongs to whoever owns the feature.

An agent waiting on a summary in a support console has perhaps four seconds of patience before they start reading the ticket themselves; past that the summary is worthless even when it arrives. A nightly batch job over yesterday's tickets can happily wait two minutes. Same model, same prompt, timeouts differing by a factor of thirty — because the number comes from the product, not the provider.

The trap is that a per-request timeout is not the one your user experiences. Three retries on a five-second timeout is a fifteen-second wait plus backoff, and the user who would not wait five seconds is certainly not waiting twenty. Set one deadline for the whole operation and give each attempt whatever remains of it.

Notice the check before sleeping. Waiting two seconds to make an attempt with one second of budget left is a guaranteed failure you paid two seconds to reach — better to give up now and spend that time on the fallback.

Backoff, jitter, and the thundering herd

Now the part that turns an incident into a longer incident. When a provider sheds load, all of your callers fail at close to the same moment. If each waits a fixed one second and tries again, they return together, in a synchronised wave, onto a service that is already struggling. That wave is the thundering herd, and it is how a thirty-second wobble becomes a ten-minute one — your own retries supplying the load that prevents recovery.

Two ingredients fix it. Exponential backoff doubles the wait after each attempt, so a caller that keeps failing keeps stepping further out of the way. Jitter randomises the wait so callers who failed together do not return together. Full jitter — picking uniformly between zero and the current window, rather than adding a wobble to a fixed delay — spreads the herd best, and is no harder to write.

Bad — every worker sleeps the same second and returns as one synchronised wave.

Good — the wait doubles, the herd spreads, and the provider's own advice wins when it is offered.

A fixed delay does not merely fail to help — during a real degradation it is the mechanism that keeps the provider down, and your traffic is part of the reason your own recovery is slow.

One more limit belongs here. Cap attempts at three or four, and cap retries as a share of total traffic — a retry budget of, say, ten percent. Per-request caps are fine when one request fails; they do nothing when every request fails, because total failure with three attempts each is three times your normal load aimed at a service that cannot handle one times.

The failures that arrive with a 200

Every rule so far keys off an exception. The two failure modes unique to model calls raise nothing at all, which is why they reach production so reliably: your error handling never runs.

A refusal is a successful response whose content declines the task. It is terminal by construction — the input that tripped the filter is exactly as objectionable on the second attempt, so a retry buys a slower, dearer copy of the same answer. Detect it (most providers give it a distinct stop reason, and an apologetic body is a decent secondary signal), stop, and hand the ticket to a human with an honest note. Looping while you rephrase the prompt to slip past the filter is both futile and the wrong instinct.

Truncation is the one that does real damage, because a fragment of a summary looks exactly like a short summary.

Bad — stores half a sentence as a finished summary, with no error anywhere.

Good — reads the stop reason before treating the body as an answer.

The cost of the first version is not a crash; it is a support agent reading "the customer was charged twice and is asking for" and closing the ticket on half the story. With JSON you often get away with it, because a cut-off object fails to parse loudly — but that is luck rather than design, and it deserts you the moment a truncated array of five items parses cleanly as three.

Choosing the fallback before you need it

Retries buy you a few seconds. When they run out something still has to be returned, and a fallback nobody decided on is whatever the exception handler happened to do — usually an empty string that flows into your database as a summary. You have four honest options, and the choice is per feature.

A weaker model keeps the feature working at lower quality: right when a rougher answer still helps, wrong when it quietly sets a lower bar nobody notices for a month. A cached answer is right when staleness is tolerable — this ticket's summary as of an hour ago is fine — and wrong the moment the value moves, which is why you would never do it for a balance. A non-AI path is the underrated one: the first two sentences of the ticket body, extracted with no model at all, are worse than a summary, infinitely better than a blank box, and they never fail. And an honest error is a real answer rather than a surrender — for anything the user acts on, "we could not summarise this ticket, the full text is below" beats a guess.

The source field is the part to copy. A degraded answer that is indistinguishable from a healthy one is a lie told to your logs, your dashboards, and the person reading it. Label every result with how it was produced, carry the label all the way to the surface, and show it wherever the difference could change what someone does next.

When the provider is having a bad hour

Everything above assumes failure is the exception. During a degradation it is the baseline, and retrying turns harmful: each request now costs a full deadline before it fails, your workers fill with calls that were never going to succeed, and the fallback that could have answered instantly arrives six seconds late — for every user at once. The feature does not degrade. It stops, and takes the worker pool with it.

The tool for this is a circuit breaker: a small piece of state that watches recent failures and, past a threshold, stops letting calls through at all.

The point of the open state is that failing in one millisecond is enormously better than failing in six seconds. Your users get the extractive summary immediately, your workers keep draining the queue, and the provider gets a break from your traffic instead of your entire retry budget. The single probe in half-open is how you notice recovery without sending the whole herd back at the first sign of life.

Two habits make the breaker effective: trip it on the failure rate over a window rather than on consecutive failures, since a bad hour is rarely a clean hundred percent, and keep one breaker per provider or model rather than one global one, so a degraded path can open while everything else keeps serving.

A cheat sheet to keep

Where to go next

You can now name any failure a model call produces, decide in one question whether a retry can change it, keep your retries from becoming the outage, and answer with something you chose.

The natural follow-on is observability for non-deterministic systems, which answers what this lesson quietly assumed: that you can tell which of these six things happened. In production you rarely have a tidy exception at hand — you have a latency graph, a support complaint, and a request from an hour ago to reconstruct through its prompts, its tool calls and every one of its retries. That lesson makes those visible, and watches for the signals that show a bad hour starting before your queue depth does.

Before that, go and find the timeout on the model call in your own codebase. If it is your HTTP client's default, it is a number nobody chose and your user's patience was never consulted. Replace it with a deadline for the whole operation, then point the client at a stub that returns 429 for thirty seconds and watch what your feature gives a user. Whatever it does on that stub, it does in production too.