The Economics of an AI Product
Unit economics per request, margin as a design constraint, abuse and capacity planning, and how cost pressure quietly reshapes architecture whether or not you plan for it.
Unit economics per request, margin as a design constraint, abuse and capacity planning, and how cost pressure quietly reshapes architecture whether or not you plan for it.
The bill arrives on the third of the month and it is three times your forecast. You open the dashboard, divide total spend by total requests, and the average cost per request is exactly the number you modelled. Nothing is broken and nobody made a mistake. Your forecast was not wrong about the average — it was wrong to use one.
By the end of this lesson you will read cost as a distribution rather than a mean, hold that distribution against what your product actually charges, put an eval number on a quality-for-margin trade instead of a hunch, defend a budget against someone who wants to empty it, and price a feature before you build it rather than after the invoice explains it to you.
Fix two placeholder rates for the whole lesson: input costs 1 unit per 1,000 tokens, output costs 5 units per 1,000 tokens. Real rates move every few months. The shape does not — output priced several times above input, both roughly linear in tokens — and the shape is what all of the arithmetic below is actually about.
Take one day of a document-summarising feature. Ten thousand requests, three populations.
requests shape units each total share
9,000 2k in, 0.4k out 4.0 36,000 26%
900 40k in, 1.5k out 47.5 42,750 31%
100 12-step loop, ~500k in 600.0 60,000 43%
--------------------------------------------------------------
10,000 mean 13.9 total 138,750 100%The median request costs 4 units. The mean costs 13.9. Ninety percent of your requests cost less than the average, which is not a paradox — it is what a heavy tail does to an average. One percent of requests are 43% of the bill; ten percent are 74% of it.
Every conclusion you draw from the mean is therefore a conclusion about a hundred requests you have never looked at. "Our cost per request is 13.9 units" describes no request that your system has ever served. Nothing costs 13.9 units. Things cost 4, or 47.5, or 600.
Three mechanisms produce that tail, and they are worth naming because they need different fixes. Input size — one user pastes a two-hundred-page contract. Loop depth — an agent that takes twelve steps instead of three, where each step carries the accumulated transcript forward, so cost grows faster than step count. Retries — a failed structured-output parse, a rate-limit backoff, a fallback to a second model, each of which pays for the same work twice or three times and succeeds, so nothing in your error rate ever shows it.
A cost distribution on its own is trivia. It becomes an economic fact when you lay it against the way your product takes money.
Say the plan sells for 3,000 units per user per month, in the same made-up currency, and usage varies the way real usage varies.
plan price 3,000 units / user / month
--------------------------------------------------------------
median user 40 calls x 6.0 = 240 margin 92%
p90 user 400 calls x 14.0 = 5,600 margin -87%
p99 user 4,000 calls x 20.0 = 80,000 margin -2,567%Now blend a thousand of them.
900 median users x 240 = 216,000 14% of cost
90 p90 users x 5,600 = 504,000 33% of cost
10 p99 users x 80,000 = 800,000 53% of cost
--------------------------------------------------------------
1,000 users cost = 1,520,000
revenue = 3,000,000 gross margin 49%Forty-nine percent looks survivable, and it is — until you sell well. Ten more customers like your ten heaviest add 30,000 in revenue and 800,000 in cost. Your best sales month is your worst margin month, and the sales team is doing exactly what you asked of them.
This is the sentence to take away: a flat price against usage that varies by two orders of magnitude is a pricing problem wearing an engineering costume. A brilliant caching layer takes that 49% to perhaps 60%. It does not change the sign of what the top ten users contribute, because their cost scales with their usage and their revenue does not scale with anything.
So pricing and architecture are one conversation. If the price can change, metering the expensive path — credits, an included allowance with a stated overage, a higher tier that exists precisely for the heavy tail — removes the problem at its source, and it is a product decision that costs you nothing to build. If the price genuinely cannot change, then the architecture has to absorb the variance itself: a hard ceiling per user per period, a cheaper path once the ceiling is hit, or a queue that makes heavy work slow instead of expensive.
There are only four real levers, and you already know all of them: shorten the context, take fewer agent steps, use a weaker model, cache harder. What the earlier lessons taught you was how each one works. What matters here is what they have in common — every one of them buys margin with quality. Sometimes the quality it spends is worth nothing, which is the happy case; caching an unchanged system prompt costs you literally no accuracy. Usually it costs something, and the only question is how much and whether you measured it.
Cost pressure applies that pressure whether or not anyone plans for it. Nobody schedules a meeting called "let us make the product worse to save money." Instead a retrieval depth drops in a config file, a step budget gets tightened during an incident and never restored, a fallback to the cheaper model starts firing on 30% of traffic because a timeout was tuned down. Six months later the product is measurably worse and no single change is responsible. That is drift, and the antidote is not discipline about cost — it is attaching a number to each trade at the moment you make it.
Bad — three quality trades in one commit, none of them measured, and no way to learn which one cost you the answers.
# retrieval + agent config
# the bill is too high, trimming everything
RETRIEVED_CHUNKS = 4 # was 12
MAX_AGENT_STEPS = 3 # was 8
SUMMARY_MODEL = "small" # was "large"Good — one trade, priced on the golden set, with the number that reverses it.
# Trade retrieval depth for margin.
# Golden set, 400 cases, three runs each:
# chunks=12 -> 0.91 exact match, 14.1 units/request
# chunks=8 -> 0.90 exact match, 10.9 units/request
# chunks=4 -> 0.78 exact match, 7.7 units/request
# 8 is the knee: one point of quality buys 23% of the
# cost. Going to 4 buys another 23% and costs twelve.
# Revert if the weekly run drops below 0.88.
RETRIEVED_CHUNKS = 8The bad version saves more money. That is why people write it. What it costs is the ability to answer the only question that will ever be asked about it — "quality dropped last quarter, what changed?" — because three variables moved at once and nobody recorded where any of them started. Whether one point of exact match is a real difference or noise is the eval harness's job, taught properly in An Eval Harness You Can Trust; this lesson only insists that a number exists to argue about.
Two habits make this stick. Put the cost number in the same report as the quality number, so a change that improves one at the expense of the other is visible in one place. And treat the knee of the curve as the interesting point: quality against cost is almost never linear, and the whole craft is finding the step where the line bends.
Availability attacks try to make your service stop. There is a cheaper attack on a metered system: denial of wallet, where the goal is not to take you down but to make you unaffordable. Every request the attacker sends costs them one HTTP call and costs you real money, and your own architecture supplies the leverage.
The asymmetry is brutal and it comes from the same heavy tail you measured earlier. A 200-token request that attaches a large document and triggers a deep agent loop costs the sender nothing and costs you 600 units. A thousand of those, sent from a script overnight, is a rounding error of effort against a bill that needs a meeting. And the attacker does not have to be an attacker: a customer's misconfigured retry loop produces an identical invoice.
Bad — limits how often a user can ask, which is not the thing that costs money.
@rate_limit(requests=100, per_seconds=3600)
async def summarise(req: SummariseRequest) -> Summary:
return await run_agent(req.document)Good — reserves the estimated spend before the call and settles the real figure after it.
@rate_limit(requests=100, per_seconds=3600)
async def summarise(req: SummariseRequest) -> Summary:
estimate = estimate_units(req.document)
async with budget.reserve(req.tenant_id, estimate) as hold:
summary = await run_agent(
req.document,
step_budget=8,
unit_ceiling=estimate * 3,
)
hold.settle(summary.units)
return summaryThe same limit of a hundred requests an hour permits 400 units of spend or 60,000 depending entirely on what those requests contain — a 150x range that the rate limiter cannot see, does not measure, and will happily wave through. Counting requests tells you about load. Counting units tells you about money, and only one of those two arrives as an invoice.
Three details make a budget real rather than decorative. Reserve before, settle after — you cannot know the true cost until the call returns, so hold an estimate up front and reconcile it, or a single request can overshoot the entire budget before anything checks. Scope it to the thing you bill — per tenant if you sell to companies, per user if you sell to people, and per API key on top, because a leaked key is a wallet attack with no attacker. Give the loop its own ceiling, in units and not only in steps, because step count and cost are not proportional once context accumulates.
Money is one ceiling. Throughput is another, and it belongs in this lesson because it constrains the same decisions. Providers meter you on requests per minute and on tokens per minute, usually per key and per organisation, and a heavy-tail workload hits the token ceiling long before the request one.
account ceiling 600,000 tokens / minute
median request 2,400 tokens -> 250 / minute
tail request 520,000 tokens -> 1 / minuteOne tail request consumes the minute's worth of 217 median ones. That is not a capacity-planning footnote — it is a product fact. It means a single user uploading a large document can starve every interactive request in the same pool, and your maximum concurrent users is set by a number in somebody else's dashboard rather than by your own machines.
Two architectural moves follow directly. Isolate the pools: separate keys or separate quota allocations for interactive traffic, background jobs and evaluation runs, so a nightly backfill cannot degrade a customer session. This is close to free and it is the single highest-value capacity decision most teams have not made. And decide in advance what happens at the ceiling: queue or shed, chosen by value rather than by arrival order. Interactive requests should shed with a clear message; background work should queue; asynchronous and batch tiers, which almost always cost less per token, should absorb everything that nobody is watching in real time.
Committed capacity deserves one paragraph of caution. Buying throughput in advance usually lowers the unit rate, and it converts a variable cost into a partly fixed one. That is excellent when you fill it and painful when traffic dips, because a commitment is a floor as well as a ceiling. Model both directions before signing, and treat the break-even utilisation as a number you monitor rather than a number you assumed once.
All of the above is diagnosis. The discipline that prevents the disease is a napkin model, built at design time, when the answer can still change the design.
cost per user per month
= actions per user per month
x model calls per action
x (input_k x input_rate + output_k x output_rate)
x tail multiplierThe tail multiplier is the one term people omit, and it is the reason forecasts come in low. It is mean divided by median from your existing distribution — 13.9 over 4.0 is roughly 3.5 for the workload above — and it converts a forecast about the typical request into a forecast about the bill. Use a conservative figure from a comparable existing feature rather than a guess.
Now price a proposal. A weekly research digest: four runs per user per month, an agent averaging six model calls per run, each call carrying 18k of input and producing 900 tokens of output.
per call 18 x 1 + 0.9 x 5 = 22.5 units
per run 22.5 x 6 calls = 135.0 units
per month 135 x 4 runs = 540.0 units
with tail 540 x 1.8 = 972.0 units
vs 3,000 plan priceOne feature consumes a third of the plan price. That is a sentence which changes a design meeting, and it costs an afternoon to produce instead of a quarter to discover.
The model also tells you which lever is worth pulling. Halving an 800-token system prompt saves 0.4 units of 22.5 — under two percent. Cutting the agent from six calls to four saves a third. When the loop dominates, tuning the prompt is rearranging deck chairs, and the fact that prompt tuning is the more comfortable work is exactly why teams do it.
So write the cost target next to the quality target, in the same spec, before either is met:
quality target >= 0.90 exact match on the 400-case set
cost target <= 12 units at p95
<= 700 units per user per monthA feature that hits quality and misses cost is not finished; it is a feature you cannot afford to launch. Finding that out while the design is still an argument is the whole point.
MEASURE
report p50, p95, p99 and total never the mean alone
watch the p99/p50 ratio tail growth = design change
attribute spend per request tenant, user, feature, model
three tail sources input size, loop depth, retries
PRICE
cost/user/month vs plan price per percentile, not blended
flat price + 100x usage spread pricing problem, not a cache problem
levers: allowance + overage metered tier, hard ceiling, queue
publish the cap silent degradation costs trust
TRADE
four levers context, steps, model, cache
each one spends quality price it on the golden set
one variable per change three at once teaches nothing
record the revert threshold drift is unpriced trades stacking
DEFEND
rate limit counts requests budgets count units
reserve estimate, settle actual scope per tenant, user and key
ceiling the loop in units step count is not proportional
expensive real customer tier it, fix it, or accept in writing
PLAN
tokens/minute binds before rpm one tail request = 217 median
isolate pools interactive / batch / eval
at the ceiling: queue or shed by value, not arrival order
committed capacity a floor as well as a ceiling
FORECAST
actions x calls x tokens x rate x tail multiplier
tail multiplier = mean / median from a real workload
pull the dominant term loop depth beats prompt trimming
ship when quality AND cost pass both targets, written togetherThat is the end of this course, and the end of the track. You started with what a language model does to a token and you finish with what a token does to a balance sheet, which is a longer distance than it sounds — most of what you learned in between is the machinery that connects the two.
There is no next course. The follow-on is your own system, and it answers something no lesson can: which of these constraints is actually binding on the thing you have already built. So here is the programme, in the order that pays fastest.
Measure the distribution. Take one week of production traffic and produce a p50, p95, p99 and total cost per request for your most-used feature. If you cannot, that gap is the first thing to fix, and it is a day of work.
Find the heavy tail and name it. Pull the twenty most expensive requests of that week and read them. You will find one of the three mechanisms — a document nobody expected, a loop that ran long, a retry storm — and you will recognise it immediately.
Price one trade. Take a single cost lever, run your golden set at two or three settings, and write the quality number and the cost number in the same comment above the config value. That comment is the difference between a decision and a drift.
Write down the budget nobody has written down. Every system already has an implicit limit on what one user may spend; it is just currently set to infinity and enforced by whoever reads the invoice. Make it a number, put it in code, and decide what happens when it fires.
Four days of work, and afterwards you will know something about your product that most teams shipping AI features genuinely do not: what it costs to serve the customer you are about to sell to. Go and find out.