Context Engineering at Scale
Long contexts degrade in ways short ones do not. Compaction, hierarchical summarisation, isolating work in sub-agents with their own context, and measuring attention loss instead of assuming it away.
Long contexts degrade in ways short ones do not. Compaction, hierarchical summarisation, isolating work in sub-agents with their own context, and measuring attention loss instead of assuming it away.
Your agent is ninety tool calls into an auth migration. At step four you told it the mobile client cannot change in this release. At step ninety it proposes changing the mobile client. Six steps before that it re-read a file it had already read twice, and described it slightly differently each time.
The window is not full. Nothing errored, no limit was reported, and the run will finish and hand you a confident plan built on a constraint it stopped holding somewhere around step sixty. By the end of this lesson you will be able to measure how your own system degrades as its context grows instead of guessing at a limit, write a compaction step with a contract about what survives it, and push a long, noisy investigation into a sub-agent whose context you throw away — knowing exactly what that costs you.
Two findings sit underneath everything in this lesson. Both reproduce across model families, and both are inconvenient.
Where the fact sits changes the answer.
Accuracy is high when a decisive fact is near the beginning or the end and measurably lower in the middle, and the dip deepens as the input grows.
The placement rule — instructions first and last — runs out of road at scale. When the context is forty documents and a hundred tool results, nearly all of it is the middle.
Adding correct material makes it worse.
Hold the task fixed, leave the decisive material exactly where it was, and add more that is correct, related, and does not contain the answer. The score falls.
Nothing was removed and nothing became wrong. There is simply more to weigh, so the right thing is weighed less.
The mechanism is not mysterious. Attention normalises across positions, so a fixed amount of weight is shared out over everything present — every token you add is a claim on the same pool. And very long sequences are the thin tail of the training distribution, so behaviour there is shaped by far fewer examples than behaviour at ordinary lengths. Neither mechanism raises an error. Both produce a slightly worse answer.
So the published window size tells you the maximum a request may contain, not the amount it should. It is a room's capacity rather than a meeting's quorum: a hall that seats two hundred does not make a two-hundred-person meeting a better way to decide anything.
Designing the Context Window named the usable portion the effective context — real, smaller than the published limit, and unmarked. At scale, add a second fact about it: it is not one number. It moves with the task. Retrieving one exact identifier out of a long input degrades early; summarising the same input tolerates far more length, because every part of the input contributes to a correct answer and nothing has to be singled out. Your ceiling is a property of a model and a job, and you have as many of them as you have jobs.
There is a third fact that makes this a scale problem rather than a tuning problem. Even a perfect allocation runs out. A migration touching two hundred files, an incident review over a week of logs, a research task with sixty sources — that work generates more material than any window holds, whatever its size. Once the job outgrows a single context, you need mechanisms that operate over time: deciding what a long run carries forward, and deciding what it never has to carry at all.
You cannot inherit someone else's ceiling, and you cannot reason your way to it. You find it the way you find anything else in this course: fix the task, vary one thing, watch the score move. The one thing you vary is context length.
Start from an evaluation set where each item has a known answer and a known decisive span — the paragraph, file, or tool result the answer actually depends on. Then build padded variants of each item at increasing multiples of the minimum context it needs, with the decisive span placed at the head, the middle and the tail of each variant.
The padding matters more than anything else in the setup. Use real material from your own corpus, topically adjacent, with the answer verified absent. Repetitive synthetic filler flatters the model badly — uniform text is easy to tune out, so you will measure a ceiling you do not have and ship against it.
import itertools
import statistics
MULTIPLES = (1, 4, 16, 64)
POSITIONS = ("head", "middle", "tail")
REPEATS = 8
def build_variant(item, multiple, position, corpus):
"""Pad an item without changing what it asks."""
budget = item.needed_tokens * (multiple - 1)
padding = take_until(corpus, budget, exclude=item.source_id)
half = len(padding) // 2
if position == "head":
blocks = [item.span] + padding
elif position == "tail":
blocks = padding + [item.span]
else:
blocks = padding[:half] + [item.span] + padding[half:]
return render(blocks, item.question)Every cell of the grid runs the identical question against the identical decisive span. The only difference between cells is how much other material is in the room, and where the decisive span sits inside it.
for multiple, position in itertools.product(MULTIPLES, POSITIONS):
scores = [
grade(run(build_variant(item, multiple, position, corpus)),
item.answer)
for item in eval_set
for _ in range(REPEATS)
]
record(multiple, position, statistics.mean(scores))REPEATS is not decoration. A single run per cell draws you a
curve whether or not one exists, because these systems vary
between identical calls. Repeat each cell until the differences
you care about are larger than the spread within a cell.
Two numbers come out of the grid, and they are the two you will actually use. The knee is the multiple at which the score starts falling — your working ceiling for this task. The position gap is the spread between head, middle and tail at a given length, which tells you how much placement is still buying you and when it has stopped. The usual shape is flat, then a knee, then a slide, with the middle position leaving first and falling furthest.
Now the decision the grid forces. If your work fits under the knee, you are done and the rest of this lesson is insurance. If it does not, no amount of prompt tuning will close the gap, because the problem is the volume of material rather than the way you asked. You have exactly two moves: carry less forward, or never carry it in the first place.
Compaction is replacing a span of accumulated context with a shorter representation engineered to preserve a named set of things. That definition earns its length by ruling out the two neighbours it gets confused with. Truncation drops the oldest material and preserves nothing on purpose. Summarisation produces readable prose, and readability is not the goal — the output of a compaction is read by a model that has to keep working, not by a person catching up.
Four categories survive a compaction, and they survive verbatim.
Constraints. Every hard boundary the run has been given or has discovered: the mobile client cannot change, customer data stays in one region, the maintenance window is Sunday. These are the cheapest thing to record and the most expensive thing to lose, because losing one does not stop the run — it silently widens what the run believes it is allowed to do.
Decisions, with their reasons. What was chosen and what was rejected, each with the reason attached on the same line. A decision without its reason gets revisited; a rejection without its reason gets proposed again fifty steps later.
Identifiers. File paths, symbol names, version strings,
ticket ids, exact numbers, quoted error text. Anything the next
step has to match character for character. Paraphrase is the
enemy here — orders-api and "the orders service" are not the
same string to a grep, and the model will confidently use the
one it has.
Open questions. What is unresolved, and what was being attempted when the span ended. Without this the run restarts its own reasoning from the top and reaches a slightly different place.
What may go is everything else, but the rule for it is sharper than "everything else": discard what has an address, keep what does not. The full text of a file has an address — the path — so throw it away and let the agent re-read it if it needs it. A tool result you can re-run has an address. A web page has a URL. But the reason a colleague rejected an approach on turn seven has no address anywhere in the world except your context; nothing fetches it back. That is the material compaction exists to protect, and it is exactly the material a fluent summary treats as throat-clearing.
Bad — asks for prose, and gets prose, at the cost of every specific in it.
COMPACT_PROMPT = """
Summarise the work so far so another engineer could pick it up.
Be concise.
"""Good — names the slots, and forbids paraphrase where paraphrase destroys the value.
COMPACT_PROMPT = """
Rewrite the work so far into these four sections. Copy paths,
symbols, versions, ids, numbers and error text exactly as they
appear; do not paraphrase them. Write "none" for an empty
section.
CONSTRAINTS: hard boundaries, stated or discovered.
DECISIONS: chosen and rejected, each with its reason.
IDENTIFIERS: files, symbols, versions and ids touched.
OPEN: unresolved questions; last thing attempted.
""""Be concise" is an instruction to drop the categories above, because to a summariser the version string, the branch name and the reason a path was ruled out all read as clutter. The compaction reports success, the agent spends the next nine steps rediscovering that the mobile client cannot change, and nothing in your logs marks the moment the constraint was lost.
You do not have to take the compaction's word for it either. The identifier rule is mechanically checkable, which makes it the one part of this you can put in code:
class CompactionLoss(Exception):
pass
def guard(before, after, extract_identifiers):
"""Refuse a compaction that dropped a known identifier."""
lost = extract_identifiers(before) - extract_identifiers(after)
if lost:
raise CompactionLoss(sorted(lost))
return afterextract_identifiers is a handful of regexes over the shapes
your domain actually uses — paths, semver strings, ticket keys,
error codes. Raising here is right: a compaction that lost three
file paths is a failure you want to see and retry, not a slightly
worse context you discover four hours later.
Long runs compact more than once, and the second compaction is where the real damage happens. If it takes the first compaction as input, you are producing a copy of a copy. Loss compounds, every pass is irreversible, and nothing downstream can tell that anything is missing — the fourth-generation summary reads exactly as fluent and confident as the first.
Memory and State Across Turns names this for chat transcripts and answers it by pinning exact material so the rolling summary never sees it. At scale, in a run that produces genuinely new information for hours, pinning inside the context is not enough, because the pinned block grows without bound and you are back where you started. The structural answer is to move the record out of the context entirely.
Keep an append-only notes file — an artifact on disk, a task record, a scratch document — and write each finding to it at the moment it is found. Context then becomes a cache over that record rather than the only copy of it. Every compaction reads the notes plus the recent tail, and the previous compaction is discarded rather than folded in.
Bad — each pass summarises the last summary, so the run drifts away from what it observed.
def compact(messages, summary):
span = messages[:-KEEP_LAST]
summary = summarise([as_note(summary)] + span)
return [as_note(summary)] + messages[-KEEP_LAST:]Good — each pass extracts from raw turns and appends to a record that lives outside the window.
def compact(messages, notes_path):
span = messages[:-KEEP_LAST]
append_notes(notes_path, extract_findings(span))
notes = read_notes(notes_path)
return [as_note(notes)] + messages[-KEEP_LAST:]By the fourth compaction the first version's summary is four generations away from anything the agent actually saw, and the constraint recorded at step seven has been rewritten four times by a model optimising for brevity. There is no copy left to check it against. The second version rewrites nothing: each line was extracted once, from the turns where it was fresh, and stays as written. It also survives a crash, which the first version does not — the notes file is a resumable run for free.
Two properties keep the notes file from becoming the problem it solves. It is append-only: existing lines are never rewritten by a model, only added to or explicitly removed by code. And it is short by construction, because each entry is a constraint, a decision with its reason, an identifier or an open question, and none of those are long. A notes file that needs summarising is a notes file that has been storing file contents.
One consequence worth planning for: compaction rewrites the front of your request, which is the part a cache was holding onto. A run that compacts often and caches naively pays for a cold prefix every time it compacts. Arranging the layout so the two do not fight is the subject of Caching Across a Model Stack.
Some work is unavoidably noisy. Finding which of four hundred files implements a behaviour. Reading a week of logs to locate the request that failed. Checking sixty sources to answer one question. The answer is a paragraph; the process to reach it is enormous, and compacting afterwards does not help, because the main context already carried all of it while the work was happening.
Context isolation is running that work in a separate context whose transcript never enters the caller's. The parent sends a question, the sub-agent gets its own window, burns whatever it needs, and returns a short result. The parent's context grows by the size of the answer rather than the size of the search.
The mental model is a stack frame. The sub-agent's locals — every file it opened, every dead end, every tool result — go out of scope at return, and only the return value survives into the caller. That is the entire benefit, and it is a large one: the parent stays under its knee no matter how much reading the question required.
Now the honest half, because isolation is a trade and it is usually sold as a free win.
The nuance is gone, and its absence is invisible. A sub-agent that found three plausible call sites and picked the most likely one returns the one it picked. The parent sees a confident answer, never sees the ambiguity, and has no way to know a judgement call was made on its behalf.
Follow-up costs a full re-run. Any question the return shape did not anticipate means launching the work again from nothing, because the context that could have answered it cheaply no longer exists. This is the real design burden: the return contract has to be decided before you know what you will want to ask.
Failures get laundered into fluency. A sub-agent that hit a timeout, retried, gave up on one directory and reported on the rest produces prose the same shape as one that succeeded completely. Unless failure is a field, it arrives as a tone.
You pay for the shared framing twice. The sub-agent needs enough of the task to be useful, so the framing is sent again, and the parent waits for a whole nested run to finish before it can continue.
The lever against three of those four is the return contract — the typed shape a sub-agent is required to return.
Bad — a narrative, which the parent must trust wholesale or re-run to check.
def investigate(question):
return sub_agent(question).final_message # free proseGood — a fixed shape, where the awkward parts have somewhere to go.
from dataclasses import dataclass, field
@dataclass
class Finding:
answer: str # three sentences, max
locations: list[str] # "path:line" to re-open
rejected: list[str] = field(default_factory=list)
unresolved: list[str] = field(default_factory=list)
failures: list[str] = field(default_factory=list)
def investigate(question):
return parse_finding(sub_agent(question, schema=Finding))The one thing the parent most needs to know — that there were
three candidates and the sub-agent chose between them — is the
first thing a fluent summary drops, and prose gives it nowhere to
live. rejected and unresolved give it a home, and locations
restores something better: an address. The parent cannot recover
the nuance, but it can re-open auth/session.py:214 for a
hundred tokens instead of re-running a four-hundred-file search.
That is the compaction rule again, doing the same job at a
different scale — discard what has an address, keep what does
not.
WHY LONG CONTEXTS GET WORSE
position the middle of a long input is retrieved worst
volume correct-but-irrelevant material still costs score
no signal nothing errors; only cost and latency move
the window a maximum a request may hold, not a target
FIND THE CURVE (belongs to a model AND a task, not a model)
fix the task and the decisive span; vary only the length
pad from your own corpus, answer verified absent
sweep 1x, 4x, 16x, 64x of the minimum context needed
place the span head / middle / tail at every length
repeat each cell until effects beat run-to-run spread
read off: the knee (your ceiling), the position gap
re-run it whenever the model behind the feature changes
COMPACTION CONTRACT - these four survive, verbatim
constraints hard boundaries, stated or discovered
decisions chosen AND rejected, each with its reason
identifiers paths, symbols, versions, ids, error text
open unresolved questions, last thing attempted
discard what has an address, keep what does not
name the slots in the prompt; forbid paraphrase
assert identifiers survive - raise, do not warn
never ask for "concise": concision drops all four
NEVER COMPACT A COMPACTION
write findings to an append-only notes file when found
extract from recent turns, never from the old summary
context becomes a cache over the record, not the record
notes stay short by construction; never summarise them
free side effect: a crashed run is resumable
SUB-AGENT AS A CONTEXT BOUNDARY (isolation, not teamwork)
use for wide noisy search whose answer is small
effect parent grows by the answer, not the search
return answer, locations, rejected, unresolved,
failures - typed, never free prose
costs nuance gone and invisible; follow-up = re-run;
failures read as tone; framing paid twice
never hand it the decision or an irreversible actionYou can now treat context length as something you measure rather than something you assume: a knee you found for your own task, a compaction step with four named categories and a mechanical check that they survived, a notes file that stops the summaries eating themselves, and a sub-agent boundary you reach for with a clear view of what it costs.
Multi-Agent Orchestration takes the question this lesson deliberately refused. Here a sub-agent was one worker with one caller, and its only job was to keep a transcript out of your window. That lesson covers what happens when there are several of them, running at once, producing results that have to be reconciled — where splitting work genuinely helps, where it is expensive theatre, and what the token bill looks like when four contexts are open at the same time.
The thing to go and do today is the smallest possible version of the sweep. Take one task you already evaluate, build four variants of each item at 1x, 4x, 16x and 64x the context it actually needs, and run them. One afternoon, one grid, and you will stop arguing about whether your context is too long — you will know the multiple at which it becomes so.