intermediate30 minby DevFox

Cost and Latency Engineering

Prompt caching, model routing, batching, streaming for perceived speed, and trimming context that earns nothing. Where the money actually goes, measured rather than assumed.

AI
Performance

The support-reply drafter works. Agents stopped rewriting the drafts three weeks ago, which is the only quality signal anyone really trusts. Then two things land in the same week: a bill several times what you modelled, and a complaint that the draft takes long enough that agents have started typing their own reply before it appears.

The instinct is to reach for the model call, because that is the part that looks expensive. It usually is not. By the end of this lesson you will know how to find the term that actually dominates your bill and your p95, which levers move it, and what each lever costs you — because every one is a trade, and the ones that look free are the ones that quietly cost quality.

Find the dominant term before you cut

You already log tokens and duration per request. What that logging does not do on its own is tell you which part of the prompt the money went to, because a usage field reports one input total for the whole thing. Break it out yourself: record the token count of each component you assembled — system prompt, tool schemas, examples, retrieved chunks, history, the user's message — alongside the total.

Then aggregate a day of it. Output tokens cost several times what input tokens cost, so weight them before comparing shares, or you will read the table wrong:

python

# Set this from your provider's own output-to-input price
# ratio. The exact number matters less than using one at all.
OUTPUT_WEIGHT = 5

def spend_by_part(records):
    """Share of one day's spend, by prompt component."""
    totals = {}
    for record in records:
        for part, tokens in record["input_tokens"].items():
            totals[part] = totals.get(part, 0) + tokens
        totals["output"] = (
            totals.get("output", 0)
            + record["output_tokens"] * OUTPUT_WEIGHT
        )
    grand_total = sum(totals.values())
    return {
        part: round(value / grand_total, 2)
        for part, value in totals.items()
    }

A first run on the drafter looks like this:

tool schemas0%

retrieved chunks0%

few-shot examples0%

output0%

system prompt0%

the customer's ticket0%

One day of the drafter's spend. The bar you would have guessed was largest is at the bottom.

A third of the bill is tool schemas: nine tools described in full on every request, when the drafting step can call exactly one of them. The customer's actual ticket — the only part that differs between requests, the thing the feature is about — is three percent.

That shape is the norm. Cost per request is usually dominated by input you stopped noticing, because input accumulates. Every element was added deliberately and individually justified, and nobody removed anything. The model call is the visible part of a bill mostly written by six months of small, reasonable additions.

Do the same for time — duration per step, not per request. The two currencies have different dominant terms surprisingly often, and we come back to that at the end.

Trim the context that earns nothing

Every element in the prompt pays rent on every request, forever. The question for each one is not "is this useful?" — everything in there is arguably useful. It is: remove it, and does the evaluation score move?

That is an ablation: take the golden set you built when you learned to evaluate the feature, run it with one component removed, and compare. Three usually come back free or nearly free:

Tool schemas the step cannot use. Attach the tools for the step you are in, not the union of every tool the feature owns. This is the single largest easy win in most codebases.
Few-shot examples that duplicate the instruction. Two examples teaching the same thing as the sentence above them earn nothing. Examples pay when they show a format or an edge case words describe badly.
Retrieved chunks past the point of usefulness. A k set to a comfortable-feeling number rather than a measured one costs on every single request.

What trimming costs you is the tail. An average score that holds can hide a slice that collapsed, so read the ablation per-slice — by ticket category, by language, by length — never as one number.

Lay the prompt out so it can be cached

Prompt caching lets a provider keep the processed form of part of your prompt and reuse it on the next request, charged at a fraction of the normal input rate. It is close to free money for any feature with a large stable preamble, which is most of them.

The mechanism dictates the one rule you need. The cache matches a prefix: it compares your prompt from the first byte and stops at the first difference, and everything after that point is fresh input. So the layout rule is stable content first, variable content last — order the prompt by rate of change, slowest first: system prompt and tool schemas, then examples, then retrieved chunks, then history, then this request.

Bad — one volatile line at the top, and nothing behind it is ever cached.

python

system = (
    f"You are a support assistant. Today is {today()}. "
    f"You are helping {agent.name} on shift {shift_id}.\n"
    + REPLY_PROCEDURES      # long, stable, rarely edited
    + TONE_GUIDE            # long, stable, rarely edited
)
messages = [{"role": "system", "content": system}]

Good — the stable block leads, and the parts that change per request follow it.

python

system = REPLY_PROCEDURES + TONE_GUIDE   # identical every time
messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": (
        f"Today is {today()}. Agent: {agent.name}, "
        f"shift {shift_id}.\n\n{ticket_text}"
    )},
]

The date line is a handful of tokens. Because it sits in front of everything, it does not cost you a handful of tokens — it costs you the cache, on every request, for as long as nobody looks. Anything else that changes those leading bytes does the same: one edited word of the system prompt, a reordered tool, a session identifier, a serializer that does not sort its keys.

Shorten the output

Output is the expensive direction: several times more per token than input, and generated one token at a time, so it is also the thing the user waits through. It is the only lever that cuts cost and latency together, which is why it is often the biggest win available. Most of the waste is not the answer — it is preamble, restated input, and reasoning nobody reads.

Bad — pays for an explanation that gets thrown away, then asks nicely for brevity.

python

INSTRUCTION = (
    "Read the ticket and the similar past tickets, explain "
    "your reasoning, then write a reply to the customer. "
    "Be concise."
)

Good — names the artifact, the limit, and what not to produce.

python

INSTRUCTION = (
    "Write the reply to the customer and nothing else. "
    "At most 120 words. No preamble, no summary of the "
    "ticket, no explanation of your reasoning."
)

"Be concise" is a preference; "at most 120 words, no preamble" is a specification, and only one of them survives contact with a long ticket. The first version also pays twice for the reasoning — once in tokens, once in seconds the agent spends watching it scroll past on the way to the part they wanted.

Two qualifications. Reasoning genuinely helps quality on some tasks, so check the eval score before deleting it; if it helps, keep it and stop displaying it. And set a maximum output length as a ceiling — not as your operating point, but so one pathological request cannot generate until something else stops it. A capped response arrives truncated mid-sentence, so handle that as a failure to retry or repair, not as an answer.

Use a weaker model where the task allows

Not every step needs your strongest model. A cheaper, weaker model is often indistinguishable on narrow, well-specified steps: classifying a ticket, extracting a product name, judging whether a retrieved chunk is relevant, rewriting one sentence. It shows its limits where the task is open-ended, where long instructions must all be obeyed at once, or where the input is adversarial.

The important part is how you decide, and the answer is not a feeling after three examples in a playground. You decide from the eval set for that step. Run both models against it, compare per-slice, and swap only when the weaker one holds on the slices you care about. If it holds everywhere, take the saving. If it holds on eighty percent and collapses on the rest, you have not found a swap — you have found a routing problem, which is real machinery, taught properly in the advanced course lesson Routing and Model Portfolios.

Measure the swap end to end, not by the per-token rate. A weaker model can be cheaper per token and more expensive per request, because it writes longer answers, needs a repair round when its output does not validate, or fails often enough that the retry path doubles the work.

Batch the work nobody is waiting for

Some of your model calls have no human attached: re-summarising documents after an ingest, backfilling tags across an archive, scoring a month of conversations, running your own evaluation suite. For these, providers offer an asynchronous batch path — you submit the whole set, results arrive within a completion window measured in hours, and the per-token cost is roughly half.

The test is simple. Is anyone waiting? A user in front of a spinner makes it interactive, and no discount is worth the wait. A nightly job, a backfill or a report is batch.

What batching costs you is a second code path. Results come back detached from the request that asked for them, so you need somewhere to put them, a way to match them up, and handling for a submission where some items succeeded and some failed. Errors surface hours later, where no user-facing retry makes sense. Treat a batch job as a data pipeline that happens to call a model, not your normal call with a flag on it.

While you are here: the cheapest request is the one you never make. Before batching a backfill, check whether each source document changed since the last run. A content hash compared before submitting removes work rather than discounting it, which beats every per-token lever in this lesson.

Streaming, which changes no cost at all

Streaming delivers tokens as they are generated instead of after the last one. The request costs precisely the same, takes the same wall-clock time, and produces the same answer. It changes one number, time to first token — and that is the number the person in front of the screen experiences.

A draft that appears after eight silent seconds feels broken. The same draft that starts writing in under a second and finishes at nine feels fast, despite taking longer. Nothing improved; the waiting became legible, which for interactive features is most of the perceived-speed problem solved.

Its costs are engineering ones. You cannot validate what you have already shown, so a response failing a schema or safety check has been on screen for two seconds by the time you know. You cannot retry invisibly, for the same reason. And anything consuming the output programmatically is now parsing a partial object.

One genuine saving hides in it: streaming lets you notice that the user cancelled or navigated away, and stop generating. On a feature where people change their mind mid-answer, that is real money.

Where the latency actually goes

Instrument a multi-step feature per step and the result is regularly uncomfortable. One request through the drafter is: embed the ticket, search the vector store, rerank the candidates, assemble the prompt, call the model, validate, store. Timed honestly, the retrieval half often costs more than the model call — and the largest span inside it is the rerank, which is a model call wearing a different name.

Three things to look for once you have the spans:

Independent steps running in series. Fetching the customer's order history and searching similar tickets have nothing to say to one another. Run them concurrently and the cost becomes the larger of the two instead of the sum.
The gap between the sum of the spans and the total. If the steps add to two seconds and the request takes three, the missing second is real — connection setup, queueing, a serialization pass over a large payload — and invisible until you subtract.
The tail, not the mean. In a serial chain the p95 is dominated by whichever step has the worst tail, even if its median is unremarkable. A dependency that is usually fast and occasionally terrible does not show up in an average at all.

The fix for a slow non-model step is almost never a model change. It is a smaller k, a precomputed embedding, an index nobody built, or a timeout with a defined fallback — which brings us to the last piece.

A budget per request, and what breaks it

Every lever above is an optimisation you did once. A budget is what keeps the property true afterwards. State it in both currencies — a cost ceiling and a latency deadline per request — then decide in advance what happens when a request would exceed it.

Bad — measures the budget after the money is spent.

python

result = run_pipeline(ticket)
if result.cost_units > BUDGET_UNITS:
    log.warning("over budget", cost=result.cost_units)
return result

Good — each step asks what is left before deciding how hard to work.

python

budget = Budget(seconds=6.0, cost_units=4000)

chunks = retrieve(ticket, k=8 if budget.seconds_left() > 4
                  else 3)

if budget.seconds_left() < 1.5:
    return acknowledgement_reply(ticket)   # honest and instant

return draft_reply(ticket, chunks,
                   max_output_tokens=budget.output_cap())

The first version is a log line describing a decision nobody made. It fires on the way out, when the money is spent and the user has already waited — and because it changes nothing, it becomes noise, and then it becomes filtered.

The order of preference when a budget runs short is: shed optional work first (a smaller k, no rerank, one pass instead of two), then degrade to a cheaper path, then return something partial and honest, and only then refuse. Never let it run silently — a loop with no ceiling is how a bad afternoon becomes a bad quarter.

A cheat sheet to keep

text

MEASURE FIRST
  spend by prompt part    weight output tokens, then rank
  latency by step         cost and time differ in dominant term
  usual top line          input you stopped noticing
  usual slow step         retrieval or a tool, not the model

LEVERS
  trim dead context       tool schemas for other steps first
  prompt caching          stable content first, variable last
  shorten output          cuts cost and latency together
  weaker model per step   narrow, well-specified steps only
  batch                   roughly half, in exchange for hours
  streaming               zero cost change, huge perceived win
  skip the call           hash the input; unchanged means done

WHAT EACH ONE COSTS YOU
  trimming                the tail; ablate per-slice, never guess
  caching                 any leading-byte edit voids the prefix
  short output            reasoning you may actually need
  weaker model            longer answers, repairs, retries
  batching                a second code path and detached errors
  streaming               cannot validate or retry what is shown

BUDGET
  state it                cost ceiling and latency deadline
  check it                before each step, not after the request
  when short              shed work, degrade, partial, refuse
  never                   log a warning and continue regardless
  also set                account spend cap and per-user limit

Where to go next

You can now find the dominant term instead of guessing at it, pull the lever that matches it, and say out loud what that lever costs. The habit that outlasts the individual techniques is the order: measure, cut the largest thing you cannot justify, re-run the eval, repeat — and stop when the next cut would buy less than it risks.

Next is Shipping an AI Feature Safely. Every change here is a quality change wearing a cost change's clothes: a trimmed prompt, a weaker model, a shed rerank. That lesson covers releasing them behind a flag, on a canary, with a kill switch and a path back — so that when a saving turns out to have cost more than you measured, you hear it from your own dashboard rather than from a customer.

The thing to do today is the breakdown. Take one day of requests, weight the output tokens, and print the share by prompt component. Then look at the top line and say why it is there. In most systems the honest answer is "nobody has looked since we added it", and that sentence is worth more than any lever in this lesson.