Memory and State Across Turns
What to carry between turns and what to drop: rolling summaries, durable facts, retrieval over past conversation, and the failure where a stale memory outranks the truth in front of it.
What to carry between turns and what to drop: rolling summaries, durable facts, retrieval over past conversation, and the failure where a stale memory outranks the truth in front of it.
A customer opens your support assistant and it greets them by name, knows they are on the team plan, and picks up the migration question they abandoned on Tuesday. It looks like the model remembers them. It remembers nothing. Every one of those details was fetched by your code and pasted into the request — and the ones your code got wrong will be stated with exactly the same confidence as the ones it got right.
By the end of this lesson you will be able to say which of three quite different things you are actually building when someone asks for "memory", compact a long conversation without gutting the parts that mattered, decide what deserves to be persisted at all, and defuse the failure this whole topic exists for: a stored fact that has gone stale and now outranks the truth sitting in front of the model.
A model request is self-contained. Turn 12 is not a continuation of turn 11; it is a brand-new request that happens to contain turns 1 through 11 in its body. Nothing is retained on the other side between calls. Statefulness lives entirely in your code, in the step that assembles the request.
It is less like a conversation and more like briefing a contractor who is sharp, well-read, and has no memory of yesterday. Every morning you hand over the same folder. Whatever is not in the folder did not happen, and whatever is in the folder is treated as current.
Two consequences fall straight out of that. Every apparent memory is a write you performed earlier and a read you performed just now, so every memory bug is a bug in one of those two steps rather than in the model. And the transcript competes for the same window budget as your instructions, your tool definitions and your retrieved material — the trade-off framed in Designing the Context Window, now with a component that grows on its own every single turn.
When a product manager asks for memory, they mean one of three mechanisms with different lifetimes, different write paths and different ways of going wrong. Keeping them separate is most of the work.
| Kind | Lives for | Written by | Fails by |
|---|---|---|---|
| Conversation state | This session | Each turn | Outgrowing budget |
| Durable facts | Until changed | A decision | Going stale |
| Retrievable history | Indefinitely | Archiving | Wrong year returns |
Conversation state is the running transcript: what has been said in this session, in order. It is the only one of the three you get for free, because you are already sending it. It is also the only one that is guaranteed current.
Durable facts are the small set of things about a user or an account that should still hold next month — a display name, a preferred region, a house style, a locale. These are written on purpose, one at a time, and are the only kind that can be wrong without anyone noticing.
Retrievable history is everything said in past sessions, archived and searched on demand rather than carried in the window. It is retrieval, applied to your own transcripts, and it uses the same machinery as retrieval over documents — the mechanics of chunking and search belong to Retrieval That Actually Retrieves and are not re-derived here.
Long sessions eventually stop fitting. You have two ways to make room: drop turns, or replace a span of turns with a shorter description of them. Dropping the oldest turns is cheap and usually catastrophic, because the oldest turns are where the task was defined. Compaction — summarizing a span of the transcript and sending the summary in its place — keeps the shape of the conversation at a fraction of the size.
The naive version summarizes everything except the last few turns. It costs more than it looks like it does.
Bad — on turn 40 the model is working from a description of the task instead of the task.
def compact(messages, keep_last=10):
older = messages[:-keep_last]
recent = messages[-keep_last:]
summary = summarize(older)
return [as_system_note(summary)] + recentGood — the turns that defined the job survive every compaction verbatim.
def compact(messages, keep_first=4, keep_last=10):
head = messages[:keep_first]
middle = messages[keep_first:-keep_last]
recent = messages[-keep_last:]
summary = summarize(middle)
return head + [as_system_note(summary)] + recentThe opening turns are where the requirements, the constraints and the definition of done were stated. Summarize them away on turn 40 and the model spends turn 41 solving a slightly different problem, competently and without any sign that it has drifted.
The pattern generalises: keep the head, keep the tail, summarize the middle. The head holds the task. The tail holds the immediate context — the last few turns are where pronouns resolve, where "that one" and "the second option" still mean something. The middle is the discursive part, and it is the part that compresses honestly.
This runs repeatedly as the session grows, which is why it is called a rolling summary: on the next compaction the previous summary is itself inside the span being summarized. That is a photocopy of a photocopy. Detail that survives three passes tends to survive forever; detail dropped once is gone, and nothing downstream knows it ever existed.
Summarization is not lossy at random. It loses the same three categories every time, because a good summarizer is optimising for readable prose and these three read as clutter.
Exact values go first
"The p99 came in at 840 ms against a 600 ms budget" becomes "latency was over budget". Numbers, identifiers, version strings, quoted error text — anything you would later want to compare against.
Proper names blur
orders-api and orders-worker both become "the orders
service", and the distinction the whole conversation turned
on is gone.
Rationale disappears
The expensive one. "We chose Postgres over the managed queue because the ops team will not run a second datastore" becomes "the team decided to use Postgres".
The decision survives; the reason does not. Twenty turns later the model helpfully suggests a managed queue, because from where it is standing that option was never considered.
The fix is not a better summarization prompt. It is to stop sending those things through the summarizer at all. Extract them structurally into a pinned ledger that is carried verbatim and never re-summarized:
Pinned facts (verbatim — never summarized)
- Region: eu-central-1. Customer data must stay in the EU.
- Rejected: managed queue service. Ops will not run a second
datastore.
- Budget ceiling: EUR 400/month, confirmed turn 6.
- Signoff: Marta Kelen, not the requester.Every line is short, every line is exact, and every decision carries its reason on the same line. This block costs a few hundred tokens and buys back the specificity that compaction takes. Append to it when a new constraint or decision appears; never rewrite it in prose.
Durable facts are the first place where this stops being a technical question. The transcript is a record of what happened and needs no justification. A durable fact is a claim your product will keep making about a person, possibly for years, so the decision to store one is a product decision made before it is an engineering one.
Three questions, and a candidate has to pass all three. Will it still be true in a month? Would the user be pleased rather than unnerved to see it come back? And if it is wrong, can they find it and correct it? A fact that fails any of them belongs in the conversation and nowhere else.
Systems that extract facts automatically fail the first question constantly, because the model cannot tell a stable preference from a passing mood.
Bad — every remark in the turn becomes a permanent property of the person who made it.
for fact in extract_facts(user_message):
memory.upsert(user_id, fact.key, fact.value)Good — the product decides in advance which keys are allowed to outlive the session.
DURABLE_KEYS = {"display_name", "preferred_region", "locale"}
for fact in extract_facts(user_message):
if fact.key in DURABLE_KEYS:
memory.upsert(user_id, fact.key, fact.value)The first version turns "I'm in a rush, keep it short" into a permanent personality setting, and the user who said it on a bad Tuesday gets terse answers in March with no idea why. An allowlist is a dozen lines and it is the difference between a profile and a landfill.
Anything you do keep also inherits every rule about personal data; the beginner course settles that in What to Never Hand a Model, and nothing here overrides it.
Retrievable history is the release valve for everything that failed the persistence test. You do not have to choose between carrying a fact forever and losing it: archive whole sessions, and search them when the current turn suggests there is something worth finding.
Two rules make the difference between this helping and hurting, and both are about time rather than search quality.
Stamp every retrieved excerpt with its date and mark it as past. An eight-month-old "we're deploying on Friday", pasted in with no date, reads as this week's plan — and the model has no way to suspect otherwise.
Then treat the excerpt as evidence about the past, not as instruction for the present. A preference someone expressed in March is a data point that they may have felt that way in March. It is not a standing order, and it does not get to overrule what they are telling you now.
Here is the failure that defines this topic, and it is worth walking through slowly because it never looks like a memory bug from the outside.
Your assistant loads a profile block before every request. It says the account is on the starter plan with a limit of three projects. Yesterday the customer upgraded to the team plan in the billing portal, and your profile store was last refreshed two weeks ago. Today they write: "I'm on the team plan now and I'm still getting a project limit error." The assistant replies that the starter plan allows three projects and offers to explain upgrading.
Nothing malfunctioned. Both statements arrived as text in the same request, and nothing in that request encoded which one was more recent. If anything the profile block reads as more authoritative: it sits near the instructions, it is formatted as a fact sheet, and the user's claim is phrased as a claim. The model is not choosing to disbelieve the customer. It has no basis on which to prefer them.
That is the whole shape of it. Staleness is invisible in a flat block of text, so you have to make it visible.
Bad — bare assertions, indistinguishable from things that are true right now.
Known about this user:
plan: starter
project_limit: 3Good — every line carries where it came from and when, under a rule about what beats what.
Stored profile — may be out of date. If the conversation
contradicts anything here, the conversation is correct.
plan: starter (billing_db, read 14 days ago)
project_limit: 3 (derived from plan, read 14 days ago)Without the date and the precedence rule the model has nothing to weigh, so it defaults to the block that looks official and tells a paying customer they are on a plan they left yesterday. That is one bad answer and one support ticket about your support system.
Three defences, in the order they are worth adding. Provenance and age on every stored fact, as above, so recency is something the model can see rather than guess. One precedence rule, stated once: the current conversation beats stored facts, and stored facts beat retrieved history. And an invalidation path — whatever changes a fact in the real world must also delete it from memory, which means the upgrade handler in the billing service is part of your memory system whether or not anyone wrote that down.
THREE MEMORIES — name which one you are building
conversation state this session only append every turn
durable facts until changed deliberate write+delete
retrieved history read on demand always dated
COMPACTION
keep the head the task, constraints, definition of done
keep the tail verbatim; pronouns resolve here
summarize the middle the discursive part compresses honestly
pin verbatim numbers, ids, decisions WITH their reason
never re-pass a pin a rolling summary photocopies itself
WHAT A SUMMARY DESTROYS — assume all three, every pass
exact values 840 ms -> "over budget"
proper names orders-api -> "the orders service"
rationale the decision survives, the why does not
BEFORE YOU PERSIST — must pass all four
true in a month? no -> leave it in the transcript
pleased to see it? no -> do not store it
user can correct it? no -> do not store it
live source exists? yes -> read it, never copy it
RENDERING STORED FACTS
label the block "may be out of date"
stamp every line source + when it was read
state precedence conversation > stored > retrieved
INVALIDATION
every write path also deletes the memories it breaks
every derived fact carries an expiry
no delete path = sediment, not memoryYou can now separate the three memories, compact a session without losing the task or the reasons behind its decisions, hold a line on what is worth persisting, and stamp stored facts so that the conversation in front of the model wins when they disagree.
Next is Failure Modes, Retries, and Fallbacks, which answers what this lesson quietly assumed: that every call returns something. Compaction is itself a model call, and it can time out, get rate-limited, or come back truncated halfway through the summary — on a conversation that is already too large to send. That lesson covers which failures are worth retrying, which need a different route, and which have to reach the user immediately.
The thing to go and try: open a long conversation you had with an assistant and scroll to the middle. Write down the three most specific things in it — a number, a name, and a decision with its reason. Now summarize that stretch in 150 words the way a summarizer would, hand back only the summary, and ask the question that depends on all three. Watching which one survives teaches more about compaction than any amount of tuning the prompt.