Designing the Context Window
Context is a budget, not a bucket. What to include, in what order, and why adding more relevant material can make answers worse rather than better.
Context is a budget, not a bucket. What to include, in what order, and why adding more relevant material can make answers worse rather than better.
The feature worked for a month. A support agent asks a question, you retrieve the four most relevant paragraphs from the internal handbook, the model answers, everyone is happy. Then someone widens the retriever to twelve paragraphs — more of the right material, surely — and the answers go vague. The prompt did not change. The model did not change. Quality dropped because you added correct information.
That is the most common surprise in production AI work, and it is not a bug in the retriever. Context is a budget you allocate, not a bucket you fill: every block you add makes every other block a smaller fraction of what the model is weighing. By the end of this lesson you will have a written allocation for your feature, an assembly order that holds up when the input is long, and a way to find out which blocks are earning their place instead of riding along.
Bucket thinking sounds reasonable: the window holds far more than you are sending, there is room, so it can go in. Free space reads as free. The trouble is that the model does not process your context the way a disk stores a file. It weighs every token against every other token, and the thing you care about — the one sentence that decides the answer — is competing with everything else you sent.
The limit that bites first is not the one the provider publishes. Call the portion of the window a model actually uses well its effective context: real, smaller than the advertised limit, and unmarked. Nothing errors when you cross it. Requests still fit comfortably, latency looks fine, and answers quietly get more generic. You find the boundary by watching quality, not by watching a counter.
So the useful number is not how many tokens fit. It is what fraction of what the model reads is the part it needs. Think of the context as a briefing you read aloud to someone who has to act the moment you stop talking. Adding a page is never free — it pushes everything else further from the moment of action, and it makes your one crucial instruction a smaller share of what they just heard.
The practical move is to pick a working budget well under the model's limit and treat that number as fixed. Once it is fixed, every new block has a price, and the price is a question: what comes out to make room?
Here is the test, and it is deliberately harsh. For each block in your prompt, name one answer that gets measurably worse if you delete it. Not "it gives helpful background" — name the input, and say how the output degrades. If you cannot name one, the block is freight: it is there because someone added it during debugging, or because the record happened to be in a variable.
The usual freight, in roughly the order I find it:
Bad — ships every field the ticket table happens to have, including the ones written for staff.
ticket = load_ticket(ticket_id)
blocks.append("Ticket:\n" + json.dumps(ticket, indent=2))Good — projects the four fields the answer depends on.
ticket = load_ticket(ticket_id)
blocks.append(
"Ticket:\n"
f"subject: {ticket['subject']}\n"
f"plan: {ticket['plan']}\n"
f"opened: {ticket['opened_at']}\n"
f"body: {ticket['body']}"
)The full record carries internal_notes, risk_score and
assigned_agent_email. Every one of those competes with the
ticket body for the model's attention, and any of them can
surface verbatim in a reply the customer reads. The projection
also fails loudly when the schema changes, instead of silently
widening.
Back to the retriever that went from four paragraphs to twelve. Three separate things happen, and it is worth naming them apart because they have different fixes.
Dilution is the arithmetic one. The decisive sentence used to be one of thirty; now it is one of four hundred. You did not weaken the signal, you raised the noise floor around it. Adding material always does this, even when the material is good.
Distraction is the near miss. Paragraph nine is about refunds — genuinely, topically about refunds — but it answers a question adjacent to the one that was asked. It looks exactly as authoritative as paragraph two, because nothing in the text says otherwise, and it pulls the answer sideways.
Contradiction is the expensive one. Two passages disagree, usually because one of them is old. The model has no mechanism for adjudicating them; it produces a fluent answer from one, or worse, an average of both, and flags nothing.
The cheap defence against distraction is a stamp on every block: where it came from, when it was last changed, and whether it is current. That gives the model something to prefer with, and it gives you something to grep for when an answer goes wrong.
<doc source="billing/refunds.md" updated="2026-04-11"
status="current">
Refunds are issued to the original payment method within ten
business days of approval...
</doc>Attention across a long input is not even. Material at the very start and the very end of the context gets weighed more heavily than material in the middle, and the effect grows with length — a well-documented result that shows up across model families. The middle of a long prompt is where things go to be skimmed.
That has a blunt consequence: the middle is for material you can afford to have skimmed. It is not where your instruction goes.
Bad — the ask is stranded tens of thousands of tokens before the answer begins.
prompt = (
"List only the policy changes that affect the Team plan,\n"
"newest first, as bullet points.\n\n"
"<policy>\n"
f"{policy_document}\n"
"</policy>"
)Good — the ask sits last, against the point where generation starts.
prompt = (
"<policy>\n"
f"{policy_document}\n"
"</policy>\n\n"
"List only the policy changes that affect the Team plan,\n"
"newest first, as bullet points."
)With the ask stranded at the top, you get a competent general summary of the policy, laid out in the document's own headings, with "Team plan" honoured loosely and "newest first" dropped entirely. The failure is intermittent, which is what makes it expensive: it survives your spot checks and shows up in the answers nobody reads closely.
The full shape follows from this. Durable framing goes first — role, rules, refusal policy — because the opening is the other strong position and because a stable prefix is the part that never has to be rebuilt between requests. The bulk material goes in the middle. The task and the output contract go last, nearest the answer. When a document is genuinely long, restating the ask after it is not redundancy, it is placement.
Wrap each block in a delimiter that says what it is, as in the stamped document above. Consistent tags do double duty: they help the model tell your instructions from the material it is reading, and they mark the line between text you wrote and text that arrived from elsewhere — a boundary this course returns to when it covers untrusted input.
Most teams meet their allocation for the first time during an incident, at the moment something overflows. Writing it down early costs an hour and changes the question from "will this fit" to "what does this displace", which is the question that keeps quality flat over a year.
A starting split for a document-answering feature, as fractions of a working budget:
Two of those lines are worth defending. Output reserve is not optional: the response is generated into the same window, so an input that consumes all of it leaves nothing to answer with. And examples are the first thing to evict, because their main job is teaching a shape — once a schema enforces that shape, worked examples are paying rent for a service you already bought elsewhere. What to keep from earlier turns, and how to compress it, is the subject of Memory and State Across Turns; here it is one line item with a share.
Now make the allocation something the code enforces:
from dataclasses import dataclass
@dataclass
class Block:
name: str
items: list[str] # chunks or turns, best or newest first
share: float # fraction of the working budget
evict: str # "oldest", "lowest", or "never"
class ContextOverflow(Exception):
pass
def fit(block, allowance, count_tokens):
kept = list(block.items)
while kept and count_tokens("\n\n".join(kept)) > allowance:
if block.evict == "never":
raise ContextOverflow(
f"{block.name} exceeds its {allowance}-token"
" allowance"
)
kept.pop(0 if block.evict == "oldest" else -1)
return "\n\n".join(kept)
def render(blocks, working_tokens, count_tokens):
return "\n\n".join(
fit(block, int(working_tokens * block.share), count_tokens)
for block in blocks
)Because chunks arrive ranked and turns arrive in order, popping
the last item drops the lowest-scoring chunk and popping the
first drops the oldest turn — the eviction rule is one word per
block. A block marked never raises instead of shrinking, which
is the behaviour you want: rules that quietly lost their last
paragraph are far worse than a request that failed loudly.
blocks = [
Block("rules", [SYSTEM_RULES], 0.10, evict="never"),
Block("tools", tool_specs, 0.05, evict="never"),
Block("examples", worked_examples, 0.10, evict="lowest"),
Block("history", turns, 0.20, evict="oldest"),
Block("documents", chunks, 0.45, evict="lowest"),
]
prompt = render(blocks, int(MODEL_LIMIT * 0.5), count_tokens)Half the model's limit is a deliberate choice, not caution for its own sake. You are buying the region where attention is reliable, and leaving room for the request that arrives with an unusually large attachment.
Bad — assembles everything, then cuts from the end when it does not fit.
prompt = "\n\n".join(blocks)
if count_tokens(prompt) > MODEL_LIMIT:
prompt = prompt[:MAX_CHARS]Good — every block owns a share, and the block that overflows is the block that shrinks.
prompt = render(blocks, int(MODEL_LIMIT * 0.5), count_tokens)The tail of a well-ordered prompt holds the task and the output contract — the two things you just spent a section putting closest to the answer. Cutting from the end deletes exactly those and leaves the bulk material intact, so the model dutifully answers a question nobody asked. Slicing characters against a token limit is a second bug hiding inside the first.
You cannot reason your way to the right allocation, because the whole problem is that plausible-looking material has non-obvious costs. You find out by removing things.
Take a fixed set of real inputs with known-good answers, run it, record the score. Then delete one block, run the same set, and compare. If the score does not move, that block was freight and you have just bought back its share of the budget. If it drops, you have a number that justifies the space instead of a feeling. Do exactly one block at a time — two at once and you cannot attribute the change.
The same method settles arguments about quantity. Run the set at
four retrieved chunks, at eight, at twelve. The curve almost
always rises and then falls, and the top of it is your k. It is
usually lower than the number someone reached for.
ASSEMBLY ORDER (first to last)
1 role and durable rules stable prefix, rarely rebuilt
2 tool definitions only tools this request can use
3 examples only while format is unfixed
4 conversation history oldest evicted first
5 retrieved material stamped: source, date, status
6 task and output contract last, closest to the answer
A STARTING SPLIT (of a working budget, not the model's limit)
rules 10% role, policy, refusal rules
tools 5%
examples 10% first to be evicted
history 20%
retrieved 45%
output reserve 10% the answer shares this window
working budget ~half the model's limit, chosen on purpose
DOES THIS BLOCK EARN ITS PLACE?
name one answer that gets worse without it
would three fields do instead of the whole record?
is it already enforced by a schema or by code?
is it here for this request, or from a debugging session?
WHY MORE RELEVANT MATERIAL CAN HURT
dilution the key line is 1 of 400, not 1 of 30
distraction near-miss passages answering a near question
contradiction two versions of a fact, resolved silently
EVICTION ORDER UNDER PRESSURE
examples, then oldest history, then lowest-scoring chunk
rules and tools raise instead of shrinking
never trim the tail: it holds the task and the contract
MEASURE
ablate one block, re-run a fixed set, keep the delta
sweep k upward until the score turns over, then stop
log per-block token counts on every request
track answer accuracy; treat retrieval recall as internalYou can now treat the context as something you allocate on purpose: a working budget under the limit, a share per block, an order that puts instructions where attention is strongest, an eviction rule decided in advance, and an ablation habit that tells you which blocks are real.
Everything here assumed you choose the contents before the request runs. Tools the Model Can Call takes the other case: material that arrives mid-request, in whatever size the tool happens to return, landing in the same budget you just planned. The allocation does not go away — it becomes something you defend during a request rather than before one.
The thing to try today is the ablation. Pick the block in your current prompt that you are least able to justify, delete it, and run your evaluation set both ways. Either the score moves and you have earned an argument for keeping it, or it does not and you have just recovered part of your budget for free. Both outcomes are worth more than the hour they cost.