advanced35 minby DevFox

Context Engineering at Scale

Long contexts degrade in ways short ones do not. Compaction, hierarchical summarisation, isolating work in sub-agents with their own context, and measuring attention loss instead of assuming it away.

Your agent is ninety tool calls into an auth migration. At step four you told it the mobile client cannot change in this release. At step ninety it proposes changing the mobile client. Six steps before that it re-read a file it had already read twice, and described it slightly differently each time.

The window is not full. Nothing errored, no limit was reported, and the run will finish and hand you a confident plan built on a constraint it stopped holding somewhere around step sixty. By the end of this lesson you will be able to measure how your own system degrades as its context grows instead of guessing at a limit, write a compaction step with a contract about what survives it, and push a long, noisy investigation into a sub-agent whose context you throw away — knowing exactly what that costs you.

A window is a capacity, not a plan

Two findings sit underneath everything in this lesson. Both reproduce across model families, and both are inconvenient.

Position sensitivity

Where the fact sits changes the answer.

Accuracy is high when a decisive fact is near the beginning or the end and measurably lower in the middle, and the dip deepens as the input grows.

The placement rule — instructions first and last — runs out of road at scale. When the context is forty documents and a hundred tool results, nearly all of it is the middle.

Volume sensitivity

Adding correct material makes it worse.

Hold the task fixed, leave the decisive material exactly where it was, and add more that is correct, related, and does not contain the answer. The score falls.

Nothing was removed and nothing became wrong. There is simply more to weigh, so the right thing is weighed less.

Two experiments, two findings, and neither one raises an error when it bites

The mechanism is not mysterious. Attention normalises across positions, so a fixed amount of weight is shared out over everything present — every token you add is a claim on the same pool. And very long sequences are the thin tail of the training distribution, so behaviour there is shaped by far fewer examples than behaviour at ordinary lengths. Neither mechanism raises an error. Both produce a slightly worse answer.

So the published window size tells you the maximum a request may contain, not the amount it should. It is a room's capacity rather than a meeting's quorum: a hall that seats two hundred does not make a two-hundred-person meeting a better way to decide anything.

Designing the Context Window named the usable portion the effective context — real, smaller than the published limit, and unmarked. At scale, add a second fact about it: it is not one number. It moves with the task. Retrieving one exact identifier out of a long input degrades early; summarising the same input tolerates far more length, because every part of the input contributes to a correct answer and nothing has to be singled out. Your ceiling is a property of a model and a job, and you have as many of them as you have jobs.

There is a third fact that makes this a scale problem rather than a tuning problem. Even a perfect allocation runs out. A migration touching two hundred files, an incident review over a week of logs, a research task with sixty sources — that work generates more material than any window holds, whatever its size. Once the job outgrows a single context, you need mechanisms that operate over time: deciding what a long run carries forward, and deciding what it never has to carry at all.

Find your own degradation curve

You cannot inherit someone else's ceiling, and you cannot reason your way to it. You find it the way you find anything else in this course: fix the task, vary one thing, watch the score move. The one thing you vary is context length.

Start from an evaluation set where each item has a known answer and a known decisive span — the paragraph, file, or tool result the answer actually depends on. Then build padded variants of each item at increasing multiples of the minimum context it needs, with the decisive span placed at the head, the middle and the tail of each variant.

The padding matters more than anything else in the setup. Use real material from your own corpus, topically adjacent, with the answer verified absent. Repetitive synthetic filler flatters the model badly — uniform text is easy to tune out, so you will measure a ceiling you do not have and ship against it.

Every cell of the grid runs the identical question against the identical decisive span. The only difference between cells is how much other material is in the room, and where the decisive span sits inside it.

REPEATS is not decoration. A single run per cell draws you a curve whether or not one exists, because these systems vary between identical calls. Repeat each cell until the differences you care about are larger than the spread within a cell.

Two numbers come out of the grid, and they are the two you will actually use. The knee is the multiple at which the score starts falling — your working ceiling for this task. The position gap is the spread between head, middle and tail at a given length, which tells you how much placement is still buying you and when it has stopped. The usual shape is flat, then a knee, then a slide, with the middle position leaving first and falling furthest.

Now the decision the grid forces. If your work fits under the knee, you are done and the rest of this lesson is insurance. If it does not, no amount of prompt tuning will close the gap, because the problem is the volume of material rather than the way you asked. You have exactly two moves: carry less forward, or never carry it in the first place.

Compaction is a transform, not a trim

Compaction is replacing a span of accumulated context with a shorter representation engineered to preserve a named set of things. That definition earns its length by ruling out the two neighbours it gets confused with. Truncation drops the oldest material and preserves nothing on purpose. Summarisation produces readable prose, and readability is not the goal — the output of a compaction is read by a model that has to keep working, not by a person catching up.

Four categories survive a compaction, and they survive verbatim.

Constraints. Every hard boundary the run has been given or has discovered: the mobile client cannot change, customer data stays in one region, the maintenance window is Sunday. These are the cheapest thing to record and the most expensive thing to lose, because losing one does not stop the run — it silently widens what the run believes it is allowed to do.

Decisions, with their reasons. What was chosen and what was rejected, each with the reason attached on the same line. A decision without its reason gets revisited; a rejection without its reason gets proposed again fifty steps later.

Identifiers. File paths, symbol names, version strings, ticket ids, exact numbers, quoted error text. Anything the next step has to match character for character. Paraphrase is the enemy here — orders-api and "the orders service" are not the same string to a grep, and the model will confidently use the one it has.

Open questions. What is unresolved, and what was being attempted when the span ended. Without this the run restarts its own reasoning from the top and reaches a slightly different place.

What may go is everything else, but the rule for it is sharper than "everything else": discard what has an address, keep what does not. The full text of a file has an address — the path — so throw it away and let the agent re-read it if it needs it. A tool result you can re-run has an address. A web page has a URL. But the reason a colleague rejected an approach on turn seven has no address anywhere in the world except your context; nothing fetches it back. That is the material compaction exists to protect, and it is exactly the material a fluent summary treats as throat-clearing.

Bad — asks for prose, and gets prose, at the cost of every specific in it.

Good — names the slots, and forbids paraphrase where paraphrase destroys the value.

"Be concise" is an instruction to drop the categories above, because to a summariser the version string, the branch name and the reason a path was ruled out all read as clutter. The compaction reports success, the agent spends the next nine steps rediscovering that the mobile client cannot change, and nothing in your logs marks the moment the constraint was lost.

You do not have to take the compaction's word for it either. The identifier rule is mechanically checkable, which makes it the one part of this you can put in code:

extract_identifiers is a handful of regexes over the shapes your domain actually uses — paths, semver strings, ticket keys, error codes. Raising here is right: a compaction that lost three file paths is a failure you want to see and retry, not a slightly worse context you discover four hours later.

Compact from the source, not the compaction

Long runs compact more than once, and the second compaction is where the real damage happens. If it takes the first compaction as input, you are producing a copy of a copy. Loss compounds, every pass is irreversible, and nothing downstream can tell that anything is missing — the fourth-generation summary reads exactly as fluent and confident as the first.

Memory and State Across Turns names this for chat transcripts and answers it by pinning exact material so the rolling summary never sees it. At scale, in a run that produces genuinely new information for hours, pinning inside the context is not enough, because the pinned block grows without bound and you are back where you started. The structural answer is to move the record out of the context entirely.

Keep an append-only notes file — an artifact on disk, a task record, a scratch document — and write each finding to it at the moment it is found. Context then becomes a cache over that record rather than the only copy of it. Every compaction reads the notes plus the recent tail, and the previous compaction is discarded rather than folded in.

Bad — each pass summarises the last summary, so the run drifts away from what it observed.

Good — each pass extracts from raw turns and appends to a record that lives outside the window.

By the fourth compaction the first version's summary is four generations away from anything the agent actually saw, and the constraint recorded at step seven has been rewritten four times by a model optimising for brevity. There is no copy left to check it against. The second version rewrites nothing: each line was extracted once, from the turns where it was fresh, and stays as written. It also survives a crash, which the first version does not — the notes file is a resumable run for free.

Two properties keep the notes file from becoming the problem it solves. It is append-only: existing lines are never rewritten by a model, only added to or explicitly removed by code. And it is short by construction, because each entry is a constraint, a decision with its reason, an identifier or an open question, and none of those are long. A notes file that needs summarising is a notes file that has been storing file contents.

One consequence worth planning for: compaction rewrites the front of your request, which is the part a cache was holding onto. A run that compacts often and caches naively pays for a cold prefix every time it compacts. Arranging the layout so the two do not fight is the subject of Caching Across a Model Stack.

Isolating a noisy investigation in a sub-agent

Some work is unavoidably noisy. Finding which of four hundred files implements a behaviour. Reading a week of logs to locate the request that failed. Checking sixty sources to answer one question. The answer is a paragraph; the process to reach it is enormous, and compacting afterwards does not help, because the main context already carried all of it while the work was happening.

Context isolation is running that work in a separate context whose transcript never enters the caller's. The parent sends a question, the sub-agent gets its own window, burns whatever it needs, and returns a short result. The parent's context grows by the size of the answer rather than the size of the search.

The mental model is a stack frame. The sub-agent's locals — every file it opened, every dead end, every tool result — go out of scope at return, and only the return value survives into the caller. That is the entire benefit, and it is a large one: the parent stays under its knee no matter how much reading the question required.

Now the honest half, because isolation is a trade and it is usually sold as a free win.

The nuance is gone, and its absence is invisible. A sub-agent that found three plausible call sites and picked the most likely one returns the one it picked. The parent sees a confident answer, never sees the ambiguity, and has no way to know a judgement call was made on its behalf.

Follow-up costs a full re-run. Any question the return shape did not anticipate means launching the work again from nothing, because the context that could have answered it cheaply no longer exists. This is the real design burden: the return contract has to be decided before you know what you will want to ask.

Failures get laundered into fluency. A sub-agent that hit a timeout, retried, gave up on one directory and reported on the rest produces prose the same shape as one that succeeded completely. Unless failure is a field, it arrives as a tone.

You pay for the shared framing twice. The sub-agent needs enough of the task to be useful, so the framing is sent again, and the parent waits for a whole nested run to finish before it can continue.

The lever against three of those four is the return contract — the typed shape a sub-agent is required to return.

Bad — a narrative, which the parent must trust wholesale or re-run to check.

Good — a fixed shape, where the awkward parts have somewhere to go.

The one thing the parent most needs to know — that there were three candidates and the sub-agent chose between them — is the first thing a fluent summary drops, and prose gives it nowhere to live. rejected and unresolved give it a home, and locations restores something better: an address. The parent cannot recover the nuance, but it can re-open auth/session.py:214 for a hundred tokens instead of re-running a four-hundred-file search. That is the compaction rule again, doing the same job at a different scale — discard what has an address, keep what does not.

A cheat sheet to keep

Where to go next

You can now treat context length as something you measure rather than something you assume: a knee you found for your own task, a compaction step with four named categories and a mechanical check that they survived, a notes file that stops the summaries eating themselves, and a sub-agent boundary you reach for with a clear view of what it costs.

Multi-Agent Orchestration takes the question this lesson deliberately refused. Here a sub-agent was one worker with one caller, and its only job was to keep a transcript out of your window. That lesson covers what happens when there are several of them, running at once, producing results that have to be reconciled — where splitting work genuinely helps, where it is expensive theatre, and what the token bill looks like when four contexts are open at the same time.

The thing to go and do today is the smallest possible version of the sweep. Take one task you already evaluate, build four variants of each item at 1x, 4x, 16x and 64x the context it actually needs, and run them. One afternoon, one grid, and you will stop arguing about whether your context is too long — you will know the multiple at which it becomes so.