advanced35 minby DevFox

The Economics of an AI Product

Unit economics per request, margin as a design constraint, abuse and capacity planning, and how cost pressure quietly reshapes architecture whether or not you plan for it.

The bill arrives on the third of the month and it is three times your forecast. You open the dashboard, divide total spend by total requests, and the average cost per request is exactly the number you modelled. Nothing is broken and nobody made a mistake. Your forecast was not wrong about the average — it was wrong to use one.

By the end of this lesson you will read cost as a distribution rather than a mean, hold that distribution against what your product actually charges, put an eval number on a quality-for-margin trade instead of a hunch, defend a budget against someone who wants to empty it, and price a feature before you build it rather than after the invoice explains it to you.

Cost per request is a distribution

Fix two placeholder rates for the whole lesson: input costs 1 unit per 1,000 tokens, output costs 5 units per 1,000 tokens. Real rates move every few months. The shape does not — output priced several times above input, both roughly linear in tokens — and the shape is what all of the arithmetic below is actually about.

Take one day of a document-summarising feature. Ten thousand requests, three populations.

text

requests  shape                     units each    total  share
   9,000  2k in, 0.4k out                  4.0   36,000    26%
     900  40k in, 1.5k out                47.5   42,750    31%
     100  12-step loop, ~500k in         600.0   60,000    43%
--------------------------------------------------------------
  10,000                    mean 13.9    total  138,750   100%

9,000 ordinary requests0%

900 large-input requests0%

100 agent loops0%

Share of the day's bill, by population. The bar on the left is 90% of your traffic.

The median request costs 4 units. The mean costs 13.9. Ninety percent of your requests cost less than the average, which is not a paradox — it is what a heavy tail does to an average. One percent of requests are 43% of the bill; ten percent are 74% of it.

Every conclusion you draw from the mean is therefore a conclusion about a hundred requests you have never looked at. "Our cost per request is 13.9 units" describes no request that your system has ever served. Nothing costs 13.9 units. Things cost 4, or 47.5, or 600.

Three mechanisms produce that tail, and they are worth naming because they need different fixes. Input size — one user pastes a two-hundred-page contract. Loop depth — an agent that takes twelve steps instead of three, where each step carries the accumulated transcript forward, so cost grows faster than step count. Retries — a failed structured-output parse, a rate-limit backoff, a fallback to a second model, each of which pays for the same work twice or three times and succeeds, so nothing in your error rate ever shows it.

Unit economics against what you charge

A cost distribution on its own is trivia. It becomes an economic fact when you lay it against the way your product takes money.

Say the plan sells for 3,000 units per user per month, in the same made-up currency, and usage varies the way real usage varies.

Now blend a thousand of them.

Forty-nine percent looks survivable, and it is — until you sell well. Ten more customers like your ten heaviest add 30,000 in revenue and 800,000 in cost. Your best sales month is your worst margin month, and the sales team is doing exactly what you asked of them.

This is the sentence to take away: a flat price against usage that varies by two orders of magnitude is a pricing problem wearing an engineering costume. A brilliant caching layer takes that 49% to perhaps 60%. It does not change the sign of what the top ten users contribute, because their cost scales with their usage and their revenue does not scale with anything.

So pricing and architecture are one conversation. If the price can change, metering the expensive path — credits, an included allowance with a stated overage, a higher tier that exists precisely for the heavy tail — removes the problem at its source, and it is a product decision that costs you nothing to build. If the price genuinely cannot change, then the architecture has to absorb the variance itself: a hard ceiling per user per period, a cheaper path once the ceiling is hit, or a queue that makes heavy work slow instead of expensive.

Every cost lever is a quality trade

There are only four real levers, and you already know all of them: shorten the context, take fewer agent steps, use a weaker model, cache harder. What the earlier lessons taught you was how each one works. What matters here is what they have in common — every one of them buys margin with quality. Sometimes the quality it spends is worth nothing, which is the happy case; caching an unchanged system prompt costs you literally no accuracy. Usually it costs something, and the only question is how much and whether you measured it.

Cost pressure applies that pressure whether or not anyone plans for it. Nobody schedules a meeting called "let us make the product worse to save money." Instead a retrieval depth drops in a config file, a step budget gets tightened during an incident and never restored, a fallback to the cheaper model starts firing on 30% of traffic because a timeout was tuned down. Six months later the product is measurably worse and no single change is responsible. That is drift, and the antidote is not discipline about cost — it is attaching a number to each trade at the moment you make it.

Bad — three quality trades in one commit, none of them measured, and no way to learn which one cost you the answers.

Good — one trade, priced on the golden set, with the number that reverses it.

The bad version saves more money. That is why people write it. What it costs is the ability to answer the only question that will ever be asked about it — "quality dropped last quarter, what changed?" — because three variables moved at once and nobody recorded where any of them started. Whether one point of exact match is a real difference or noise is the eval harness's job, taught properly in An Eval Harness You Can Trust; this lesson only insists that a number exists to argue about.

Two habits make this stick. Put the cost number in the same report as the quality number, so a change that improves one at the expense of the other is visible in one place. And treat the knee of the curve as the interesting point: quality against cost is almost never linear, and the whole craft is finding the step where the line bends.

Denial of wallet

Availability attacks try to make your service stop. There is a cheaper attack on a metered system: denial of wallet, where the goal is not to take you down but to make you unaffordable. Every request the attacker sends costs them one HTTP call and costs you real money, and your own architecture supplies the leverage.

The asymmetry is brutal and it comes from the same heavy tail you measured earlier. A 200-token request that attaches a large document and triggers a deep agent loop costs the sender nothing and costs you 600 units. A thousand of those, sent from a script overnight, is a rounding error of effort against a bill that needs a meeting. And the attacker does not have to be an attacker: a customer's misconfigured retry loop produces an identical invoice.

Bad — limits how often a user can ask, which is not the thing that costs money.

Good — reserves the estimated spend before the call and settles the real figure after it.

The same limit of a hundred requests an hour permits 400 units of spend or 60,000 depending entirely on what those requests contain — a 150x range that the rate limiter cannot see, does not measure, and will happily wave through. Counting requests tells you about load. Counting units tells you about money, and only one of those two arrives as an invoice.

Three details make a budget real rather than decorative. Reserve before, settle after — you cannot know the true cost until the call returns, so hold an estimate up front and reconcile it, or a single request can overshoot the entire budget before anything checks. Scope it to the thing you bill — per tenant if you sell to companies, per user if you sell to people, and per API key on top, because a leaked key is a wallet attack with no attacker. Give the loop its own ceiling, in units and not only in steps, because step count and cost are not proportional once context accumulates.

Capacity is a product constraint

Money is one ceiling. Throughput is another, and it belongs in this lesson because it constrains the same decisions. Providers meter you on requests per minute and on tokens per minute, usually per key and per organisation, and a heavy-tail workload hits the token ceiling long before the request one.

One tail request consumes the minute's worth of 217 median ones. That is not a capacity-planning footnote — it is a product fact. It means a single user uploading a large document can starve every interactive request in the same pool, and your maximum concurrent users is set by a number in somebody else's dashboard rather than by your own machines.

Two architectural moves follow directly. Isolate the pools: separate keys or separate quota allocations for interactive traffic, background jobs and evaluation runs, so a nightly backfill cannot degrade a customer session. This is close to free and it is the single highest-value capacity decision most teams have not made. And decide in advance what happens at the ceiling: queue or shed, chosen by value rather than by arrival order. Interactive requests should shed with a clear message; background work should queue; asynchronous and batch tiers, which almost always cost less per token, should absorb everything that nobody is watching in real time.

Committed capacity deserves one paragraph of caution. Buying throughput in advance usually lowers the unit rate, and it converts a variable cost into a partly fixed one. That is excellent when you fill it and painful when traffic dips, because a commitment is a floor as well as a ceiling. Model both directions before signing, and treat the break-even utilisation as a number you monitor rather than a number you assumed once.

Forecasting a feature before you build it

All of the above is diagnosis. The discipline that prevents the disease is a napkin model, built at design time, when the answer can still change the design.

The tail multiplier is the one term people omit, and it is the reason forecasts come in low. It is mean divided by median from your existing distribution — 13.9 over 4.0 is roughly 3.5 for the workload above — and it converts a forecast about the typical request into a forecast about the bill. Use a conservative figure from a comparable existing feature rather than a guess.

Now price a proposal. A weekly research digest: four runs per user per month, an agent averaging six model calls per run, each call carrying 18k of input and producing 900 tokens of output.

One feature consumes a third of the plan price. That is a sentence which changes a design meeting, and it costs an afternoon to produce instead of a quarter to discover.

The model also tells you which lever is worth pulling. Halving an 800-token system prompt saves 0.4 units of 22.5 — under two percent. Cutting the agent from six calls to four saves a third. When the loop dominates, tuning the prompt is rearranging deck chairs, and the fact that prompt tuning is the more comfortable work is exactly why teams do it.

So write the cost target next to the quality target, in the same spec, before either is met:

A feature that hits quality and misses cost is not finished; it is a feature you cannot afford to launch. Finding that out while the design is still an argument is the whole point.

A cheat sheet to keep

Where to go next

That is the end of this course, and the end of the track. You started with what a language model does to a token and you finish with what a token does to a balance sheet, which is a longer distance than it sounds — most of what you learned in between is the machinery that connects the two.

There is no next course. The follow-on is your own system, and it answers something no lesson can: which of these constraints is actually binding on the thing you have already built. So here is the programme, in the order that pays fastest.

Measure the distribution. Take one week of production traffic and produce a p50, p95, p99 and total cost per request for your most-used feature. If you cannot, that gap is the first thing to fix, and it is a day of work.

Find the heavy tail and name it. Pull the twenty most expensive requests of that week and read them. You will find one of the three mechanisms — a document nobody expected, a loop that ran long, a retry storm — and you will recognise it immediately.

Price one trade. Take a single cost lever, run your golden set at two or three settings, and write the quality number and the cost number in the same comment above the config value. That comment is the difference between a decision and a drift.

Write down the budget nobody has written down. Every system already has an implicit limit on what one user may spend; it is just currently set to infinity and enforced by whoever reads the invoice. Make it a number, put it in code, and decide what happens when it fires.

Four days of work, and afterwards you will know something about your product that most teams shipping AI features genuinely do not: what it costs to serve the customer you are about to sell to. Go and find out.