intermediate30 minby DevFox

Memory and State Across Turns

What to carry between turns and what to drop: rolling summaries, durable facts, retrieval over past conversation, and the failure where a stale memory outranks the truth in front of it.

A customer opens your support assistant and it greets them by name, knows they are on the team plan, and picks up the migration question they abandoned on Tuesday. It looks like the model remembers them. It remembers nothing. Every one of those details was fetched by your code and pasted into the request — and the ones your code got wrong will be stated with exactly the same confidence as the ones it got right.

By the end of this lesson you will be able to say which of three quite different things you are actually building when someone asks for "memory", compact a long conversation without gutting the parts that mattered, decide what deserves to be persisted at all, and defuse the failure this whole topic exists for: a stored fact that has gone stale and now outranks the truth sitting in front of the model.

Nothing carries over but what you resend

A model request is self-contained. Turn 12 is not a continuation of turn 11; it is a brand-new request that happens to contain turns 1 through 11 in its body. Nothing is retained on the other side between calls. Statefulness lives entirely in your code, in the step that assembles the request.

It is less like a conversation and more like briefing a contractor who is sharp, well-read, and has no memory of yesterday. Every morning you hand over the same folder. Whatever is not in the folder did not happen, and whatever is in the folder is treated as current.

Two consequences fall straight out of that. Every apparent memory is a write you performed earlier and a read you performed just now, so every memory bug is a bug in one of those two steps rather than in the model. And the transcript competes for the same window budget as your instructions, your tool definitions and your retrieved material — the trade-off framed in Designing the Context Window, now with a component that grows on its own every single turn.

Three different things called memory

When a product manager asks for memory, they mean one of three mechanisms with different lifetimes, different write paths and different ways of going wrong. Keeping them separate is most of the work.

Kind	Lives for	Written by	Fails by
Conversation state	This session	Each turn	Outgrowing budget
Durable facts	Until changed	A decision	Going stale
Retrievable history	Indefinitely	Archiving	Wrong year returns

Conversation state is the running transcript: what has been said in this session, in order. It is the only one of the three you get for free, because you are already sending it. It is also the only one that is guaranteed current.

Durable facts are the small set of things about a user or an account that should still hold next month — a display name, a preferred region, a house style, a locale. These are written on purpose, one at a time, and are the only kind that can be wrong without anyone noticing.

Retrievable history is everything said in past sessions, archived and searched on demand rather than carried in the window. It is retrieval, applied to your own transcripts, and it uses the same machinery as retrieval over documents — the mechanics of chunking and search belong to Retrieval That Actually Retrieves and are not re-derived here.

Compacting a conversation without gutting it

Long sessions eventually stop fitting. You have two ways to make room: drop turns, or replace a span of turns with a shorter description of them. Dropping the oldest turns is cheap and usually catastrophic, because the oldest turns are where the task was defined. Compaction — summarizing a span of the transcript and sending the summary in its place — keeps the shape of the conversation at a fraction of the size.

The naive version summarizes everything except the last few turns. It costs more than it looks like it does.

Bad — on turn 40 the model is working from a description of the task instead of the task.

Good — the turns that defined the job survive every compaction verbatim.

The opening turns are where the requirements, the constraints and the definition of done were stated. Summarize them away on turn 40 and the model spends turn 41 solving a slightly different problem, competently and without any sign that it has drifted.

The pattern generalises: keep the head, keep the tail, summarize the middle. The head holds the task. The tail holds the immediate context — the last few turns are where pronouns resolve, where "that one" and "the second option" still mean something. The middle is the discursive part, and it is the part that compresses honestly.

This runs repeatedly as the session grows, which is why it is called a rolling summary: on the next compaction the previous summary is itself inside the span being summarized. That is a photocopy of a photocopy. Detail that survives three passes tends to survive forever; detail dropped once is gone, and nothing downstream knows it ever existed.

What a summary reliably destroys

Summarization is not lossy at random. It loses the same three categories every time, because a good summarizer is optimising for readable prose and these three read as clutter.

Exact values go first
"The p99 came in at 840 ms against a 600 ms budget" becomes "latency was over budget". Numbers, identifiers, version strings, quoted error text — anything you would later want to compare against.
Proper names blur
orders-api and orders-worker both become "the orders service", and the distinction the whole conversation turned on is gone.
Rationale disappears
The expensive one. "We chose Postgres over the managed queue because the ops team will not run a second datastore" becomes "the team decided to use Postgres".
The decision survives; the reason does not. Twenty turns later the model helpfully suggests a managed queue, because from where it is standing that option was never considered.

A good summarizer is optimising for readable prose, and all three of these read as clutter

The fix is not a better summarization prompt. It is to stop sending those things through the summarizer at all. Extract them structurally into a pinned ledger that is carried verbatim and never re-summarized:

Every line is short, every line is exact, and every decision carries its reason on the same line. This block costs a few hundred tokens and buys back the specificity that compaction takes. Append to it when a new constraint or decision appears; never rewrite it in prose.

Deciding what deserves to be persisted

Durable facts are the first place where this stops being a technical question. The transcript is a record of what happened and needs no justification. A durable fact is a claim your product will keep making about a person, possibly for years, so the decision to store one is a product decision made before it is an engineering one.

Three questions, and a candidate has to pass all three. Will it still be true in a month? Would the user be pleased rather than unnerved to see it come back? And if it is wrong, can they find it and correct it? A fact that fails any of them belongs in the conversation and nowhere else.

Systems that extract facts automatically fail the first question constantly, because the model cannot tell a stable preference from a passing mood.

Bad — every remark in the turn becomes a permanent property of the person who made it.

Good — the product decides in advance which keys are allowed to outlive the session.

The first version turns "I'm in a rush, keep it short" into a permanent personality setting, and the user who said it on a bad Tuesday gets terse answers in March with no idea why. An allowlist is a dozen lines and it is the difference between a profile and a landfill.

Anything you do keep also inherits every rule about personal data; the beginner course settles that in What to Never Hand a Model, and nothing here overrides it.

Reaching back into old conversations

Retrievable history is the release valve for everything that failed the persistence test. You do not have to choose between carrying a fact forever and losing it: archive whole sessions, and search them when the current turn suggests there is something worth finding.

Two rules make the difference between this helping and hurting, and both are about time rather than search quality.

Stamp every retrieved excerpt with its date and mark it as past. An eight-month-old "we're deploying on Friday", pasted in with no date, reads as this week's plan — and the model has no way to suspect otherwise.

Then treat the excerpt as evidence about the past, not as instruction for the present. A preference someone expressed in March is a data point that they may have felt that way in March. It is not a standing order, and it does not get to overrule what they are telling you now.

When a stored fact outranks the truth

Here is the failure that defines this topic, and it is worth walking through slowly because it never looks like a memory bug from the outside.

Your assistant loads a profile block before every request. It says the account is on the starter plan with a limit of three projects. Yesterday the customer upgraded to the team plan in the billing portal, and your profile store was last refreshed two weeks ago. Today they write: "I'm on the team plan now and I'm still getting a project limit error." The assistant replies that the starter plan allows three projects and offers to explain upgrading.

Nothing malfunctioned. Both statements arrived as text in the same request, and nothing in that request encoded which one was more recent. If anything the profile block reads as more authoritative: it sits near the instructions, it is formatted as a fact sheet, and the user's claim is phrased as a claim. The model is not choosing to disbelieve the customer. It has no basis on which to prefer them.

That is the whole shape of it. Staleness is invisible in a flat block of text, so you have to make it visible.

Bad — bare assertions, indistinguishable from things that are true right now.

Good — every line carries where it came from and when, under a rule about what beats what.

Without the date and the precedence rule the model has nothing to weigh, so it defaults to the block that looks official and tells a paying customer they are on a plan they left yesterday. That is one bad answer and one support ticket about your support system.

Three defences, in the order they are worth adding. Provenance and age on every stored fact, as above, so recency is something the model can see rather than guess. One precedence rule, stated once: the current conversation beats stored facts, and stored facts beat retrieved history. And an invalidation path — whatever changes a fact in the real world must also delete it from memory, which means the upgrade handler in the billing service is part of your memory system whether or not anyone wrote that down.

A cheat sheet to keep

Where to go next

You can now separate the three memories, compact a session without losing the task or the reasons behind its decisions, hold a line on what is worth persisting, and stamp stored facts so that the conversation in front of the model wins when they disagree.

Next is Failure Modes, Retries, and Fallbacks, which answers what this lesson quietly assumed: that every call returns something. Compaction is itself a model call, and it can time out, get rate-limited, or come back truncated halfway through the summary — on a conversation that is already too large to send. That lesson covers which failures are worth retrying, which need a different route, and which have to reach the user immediately.

The thing to go and try: open a long conversation you had with an assistant and scroll to the middle. Write down the three most specific things in it — a number, a name, and a decision with its reason. Now summarize that stretch in 150 words the way a summarizer would, hand back only the summary, and ask the question that depends on all three. Watching which one survives teaches more about compaction than any amount of tuning the prompt.