beginner25 minby DevFox

Tokens, Context, and Why They Cost You

How text becomes tokens, what a context window really limits, and why input and output are priced and paced differently. Counting tokens before they surprise you, and what happens when you run out of room.

Your summariser works. You pasted one support ticket into it, the reply came back in a couple of seconds, and the call cost a fraction of a cent. So you point it at the real queue — and three things go wrong at once. One customer's thread comes back as an error about length. Another gets a summary that stops mid-word. And the bill at the end of the month is not a fraction of anything.

None of that is bad luck. All three come from the same unit of measurement — the token — and from one fixed-size space that everything you send and everything you get back has to share. By the end of this lesson you will know how your text becomes tokens, why a character limit is not a token limit, what a context window actually holds, why the answer costs more than the question, and how to count all of it before it counts you.

The unit the model actually counts

A model cannot read your text the way you do. It needs your writing turned into a sequence of numbered pieces from a fixed vocabulary, and someone had to decide how big a piece is.

Single letters would be a terrible choice: a paragraph would become thousands of pieces, each carrying almost no meaning. Whole words are no better, because there is no finite list of words — people invent names, mistype things, paste error codes, and write in other languages. So the pieces sit in between. A token is a chunk of text, usually a few characters long: common words are a single token, and rarer words get broken into fragments the model has seen before. The program that does the chopping is the tokenizer, and it is fixed in advance — the same text always splits the same way.

Here is roughly what that looks like. The exact split depends on which tokenizer a model uses, but the shape is always this:

text

"strawberry"     ->  straw | berry
"unhelpfulness"  ->  un | help | ful | ness
"Kowalczyk"      ->  Kow | al | czyk
"1,000,000"      ->  1 | , | 000 | , | 000
" the"           ->  one token, leading space included

That last line matters more than it looks. Spaces are not free and they are not separate — they usually ride along at the front of the following token. Which is why the model sees text the way a typesetter does, not the way a dictionary does.

Where token boundaries surprise you

For ordinary English prose there is a rule of thumb worth memorising: one token is about four characters, or roughly three-quarters of a word. A 500-word email is somewhere near 650 tokens. That estimate is good enough for planning and wrong often enough that you should never build logic on it.

It drifts as soon as your text stops being ordinary English prose, and it drifts in one direction: everything else costs more.

English prose

About four characters per token.

The case the rule of thumb was built from, and the only one it is reliable for.

Code and JSON

Much denser.

Every brace, quote, colon and level of indentation is its own token. Numbers split in ways that follow no arithmetic logic at all — one reason models are shaky at mental maths.

Identifiers

Worst per unit of meaning.

A UUID or an API key is a run of characters the tokenizer has never seen as a unit, so it becomes a pile of one- and two-character tokens for a single meaningless value.

Non-Latin scripts

Several times more, for the same sentence.

The tokenizer's vocabulary was built mostly from English. Same meaning, several times the cost and several times the room used.

If your product has customers outside your own language, this is a bug waiting in your length checks.

The same character count, four very different token counts

Bad — trims to a size the model does not measure in, so the limit means something different for every customer.

Good — trims in the unit that is actually limited.

Six thousand characters of English is around 1,500 tokens, so the first version looks correct on every ticket you tested it with. The same six thousand characters of Japanese, or of minified JSON, can be several times that — so the check that passed your tests overflows in production, for exactly the customers you did not test with.

The context window is a desk, not a memory

The context window is the maximum number of tokens a model can have in front of it for a single request. It is a hard ceiling, it varies by model, and — this is the part that catches people — everything shares it.

Picture a whiteboard of fixed size. Before you ask anything, someone has already written your standing instructions on it. The whole conversation so far goes on next, every turn of it, yours and the model's. Then the document you pasted, and the descriptions of any tools the model can call. Whatever board is left is where the model has to write its answer. Every word you add takes room away from the reply.

The second half of the metaphor is the one that surprises people: the board is wiped between calls. A model has no memory of your last request. When a chat app appears to remember what you said ten minutes ago, it is because the app re-sent all of it, invisibly, as part of this request. That is a design you should find reassuring — nothing leaks between your users by accident — but it has a price tag attached. On turn ten you are paying to send turns one through nine again, and you will pay for them again on turn eleven. A long conversation gets more expensive per turn as it goes, and it is still the same conversation.

What running out of room looks like

Overflowing the window is not one failure, it is four, and only the first one is honest about itself.

The provider rejects the request
An error naming the limit and the size you sent. Loud, immediate, and it tells you the number. This is the good case.
Something silently trims your input
Plenty of libraries cut the payload down until it fits. The call succeeds and the answer looks fine — but you do not control which end got cut, and if it took the front, it took your instructions with it.
The chat layer drops the oldest turns
The conversation keeps working, and the assistant quietly stops following the rule you set at the beginning, because that message is no longer being sent.
This is the real explanation behind almost every "the model forgot what I told it".
The answer hits the ceiling and stops
Not an error — a complete, successful response that happens to end mid-sentence or, far more painfully, mid-JSON.

Three of these return a successful response

Bad — finds out the conversation is too long from the provider, in front of the user.

Good — finds out first, in your own code, with your own error.

The arithmetic was available to you before you sent anything. The first version turns a number you could have checked into a provider error mid-sentence — or, if the library trims quietly instead of failing, into an assistant that changes behaviour for reasons nobody on your team can reproduce.

Input and output are not the same currency

Tokens going in and tokens coming out are counted separately, and you should think of them as two different resources.

Output tokens cost several times more than input tokens, at every provider, and the ratio is large enough to change your design. The reason is mechanical: the model reads your entire input in one pass, but it produces the answer one token at a time, and each of those tokens requires another full pass over everything written so far. Reading is bulk work. Writing is serial work.

That same fact sets your latency. How long a call takes depends mostly on how long the answer is, not how long the question is. A large document with a one-line verdict comes back quickly. A one-line question that triggers a thousand-word essay does not. Streaming the response changes when the user sees the first word; it does not change when the last one arrives.

Three practical consequences follow. Asking for a shorter answer makes a feature both cheaper and faster, and it is usually the single biggest lever you have. Asking the model to repeat your input back to you — restating the ticket before summarising it — means paying for the same text twice, the second time at the expensive rate. And any instruction that makes the model work through its reasoning in the reply is buying real output tokens, which may well be worth it, but it is not free thinking.

Counting them before they count you

You now know the failure modes. The habit that prevents all of them is unglamorous: count, early, on real data.

Use the four-characters-per-token rule when you are sketching on paper. Use the actual tokenizer whenever a number decides something — whether to send, whether to trim, whether to warn. Every provider ships one, either as a local library or as a counting endpoint, and it is the same tokenizer their billing uses.

Count your worst realistic input, not your demo one. The ticket you tested with is not the customer who pasted forty replies and a stack trace. If you have production data, take the longest few percent and measure those; that is the input that decides whether your feature works.

Then decide your reply room deliberately. The space for the answer is a budget you set before you fill the rest, not whatever happens to be left over.

Finally, make the counts visible after the fact. Every response carries the tokens it actually used, and logging those two numbers turns "what does this feature cost us" from an argument into a query:

Log it on day one. The week you need it is the week the bill doubles and nobody can say which feature did it.

A cheat sheet to keep

Where to go next

You can now read a model's constraints in its own units: text becomes tokens on boundaries that ignore your words, one window holds the instructions, the history, your data and the answer together, and the answer is the expensive half in both money and time. That is enough to size a feature and to recognise a truncation for what it is instead of chasing it as a bug.

Next comes Sampling, Temperature, and Non-Determinism, which answers a question this lesson deliberately left alone. You now know how many tokens come back and what they cost — that lesson is about why the same request can hand you different tokens each time you run it, and how to decide whether that variation is a feature you want or a problem you have to design around. Later, once you are running something real, an intermediate course picks up what to do when the budget genuinely will not fit.

The thing to go and do right now takes ten minutes. Take the longest real input your feature will ever see, run it through your provider's tokenizer, and compare the true count against len(text) / 4. Then paste in a translation of the same text into a language that does not use the Latin alphabet, and count again. The gap between those three numbers is the one that will otherwise find you in production.