Two problems were left open at the end of the previous part: the trajectory only grows and eventually will not fit, and when the program exits it remembers nothing.

Both describe the same thing — memory. The intuitive assumption is that memory is a feature of the model, something you switch on. That assumption is wrong. The model's parameters are frozen after training, and every request arrives with no prior state. Everything called memory is built on top of the context window.

That produces two mechanisms, and only two:

  • The agent remembers earlier turns in a conversation because those turns are still inside the context.
  • The agent remembers across sessions because something was written to storage outside the program and read back into the context at startup.

This part implements both, and the toy agent reaches version 0.4.

1. The Window Is Finite, and It Degrades Before It Fills

A model can attend to a bounded number of tokens per call — the context window. Depending on the model, this ranges from a few thousand to about a million. Exceed it and the request fails outright.

Staying under the limit is not sufficient. As context volume grows, the model becomes more likely to drop individual items, and content positioned in the middle is the most likely to be lost. This is not a hardware constraint; it is a behavioral one, and it has a name: context rot. The human analogue is familiar — reading a hundred pages of material before answering usually produces worse answers than reading the three relevant pages.

The practical conclusion: do not wait for the error. Set a budget well below the hard limit, and act as soon as the trajectory crosses it.

The difference between the two approaches is visible in one picture. Left alone, content grows in a straight line until the request fails. With a budget, the line climbs to a threshold, drops back, climbs again — a sawtooth that never leaves the safe band.

Three reasons to control context length, not one:

  1. Avoiding hard failures.
  2. Preserving answer accuracy.
  3. Controlling cost.

Point three connects to an earlier calculation: the longer the context, the more expensive every turn. Prompt caching helps only when the prefix is stable, and any edit to earlier content invalidates the cached portion from that point forward. This is why compression strategy and caching strategy cannot be designed independently — a decision covered later in this piece.

2. Three Exits

When the trajectory is too long, three moves are available. They differ in where the content goes, and each carries a different cost.

Exit What happens Cost
Trim Return less at the source; elide old tool results May cut something needed later
Compress Replace older turns with a summary One extra model call; detail loss; cache invalidation
Offload Write content to a file, leave a pointer in context Must re-read when detail is needed

2.1 Trim: return less to begin with

Most context bloat comes from tool results, so the cheapest fix is at the source: tools should not return so much in the first place.

  • A note-reading tool returns only the first two thousand characters, with a trailing note that more remains and can be read on request.
  • A search tool returns only the matching lines, not the whole file.
  • Old tool results already in the trajectory are replaced with a single line: content elided, re-read if needed.

The cost is real: trimming may discard something needed later. So every trim must leave a path back — the agent has to know the content can be retrieved again.

2.2 Compress: replace old turns with a summary

Hand the earlier portion of the trajectory to the model, ask for a short summary, then substitute the summary for the original text. After compression the trajectory shrinks substantially — and anything the summary omitted is genuinely gone.

A summary must retain at least four things:

  1. The requirements the user stated.
  2. What has already been done.
  3. The conclusions reached.
  4. What remains unfinished.

Drop any one of those four and subsequent turns are likely to drift.

Compression carries three costs.

An extra model call. Every compression round costs a full request.

Detail loss. Specifics that seemed incidental at compression time often turn out to be load-bearing.

Cache invalidation. This is the one most often missed. Compression rewrites earlier content, so from the point of the summary onward, the cached prefix is invalid. Every token after that point must be reprocessed at full price.

The third cost dictates the compression schedule. You cannot compress a little on every turn — that invalidates the cache every turn, and you pay the reprocessing cost continuously. Instead, compress a large chunk once the budget is crossed. That is exactly why the sawtooth in the earlier figure drops in large steps rather than small ones.

For teams running agents across several model backends, this cost model is worth centralizing rather than rediscovering per project. Prompt cache pricing differs by provider, and the same compaction policy can be cheap on one endpoint and expensive on another. 4sapi is an API gateway that keeps multiple model endpoints behind one routing layer, which makes per-turn cost visible in one place instead of scattered across dashboards — useful when the compaction threshold itself is a cost decision.

2.3 Offload: move content outside

Write the content out of the context and leave one pointer sentence behind.

A concrete case: the agent reads ten notes and produces a consolidated summary. That summary does not need to stay in the trajectory. Write it to summary.md, and leave one line in context — summary written to summary.md. When detail is needed later, read the file.

Files have almost no capacity limit, which the context window does not. This is also why long tasks work better when the agent records progress and conclusions to a file as it goes: even if the trajectory is later compressed, the file survives.

The three exits are not mutually exclusive. Combined, they look like this:

Exit Method Cost
Trim Tools return less; old results elided May cut something still needed
Compress Old trajectory replaced by a summary Extra call; detail loss; cache invalidation
Offload Content written to file, pointer remains Extra read when detail is needed

3. The Compaction Code

The interesting part of the implementation is how compact splits the trajectory into "earlier" and "most recent" segments.

import json, os, pathlib from openai import OpenAI client = OpenAI(base_uri=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"]) MODEL = os.environ["LLM_MODEL"] BUDGET = 600 # token budget for the trajectory; small here for demonstration def as_dict(m): return m if isinstance(m, dict) else m.model_dump(exclude_none=True) def compact(messages): """Compress earlier turns into a summary when the trajectory exceeds budget.""" history = [as_dict(m) for m in messages[1:]] last_user = max((i for i, m in enumerate(history) if m["role"] == "user"), default=0) old, recent = history[:last_user], history[last_user:] if len(json.dumps(history, ensure_ascii=False)) <= BUDGET or not old: return messages ask = ("Below is an earlier segment of a conversation. Write a short summary. " "Keep: what the user asked for, what has been done, conclusions reached, work not yet finished.\n\n" + json.dumps(old, ensure_ascii=False)) summary = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": ask}]).choices[0].message.content print(f"[compact] folded {len(old)} earlier messages into one summary") return [messages[0], {"role": "user", "content": f"[summary of earlier conversation]\n" + summary}] + recent 

Three implementation details carry the design.

The split point is the last user message. Everything before it is "earlier" and gets summarized; everything from it onward is the most recent round and stays verbatim. Recent content is the most likely to be needed immediately, so it is the last thing to touch.

Summarization is a separate model call, outside the main loop. It has no tools and no agent state. The instruction explicitly names the four items to retain, because an unconstrained "summarize this" reliably drops one of them.

The returned list is: system prompt, summary, most recent round. The system prompt stays untouched at the front, so the static prefix remains byte-identical and its cache survives. Only content after the summary is reprocessed.

That last point is the whole reason the structure looks the way it does. A different ordering — summary first, system prompt after — would invalidate the entire prefix on every compression.

4. Cross-Session Memory: One File

Compression solves the problem of not fitting within a single run. But the trajectory lives only inside that run; exit the program and it is gone. To make the agent remember tomorrow, something must be stored outside the process. The simplest storage is a file.

The arrangement has two actions and no more:

  • Write. When something worth keeping long-term appears in conversation, the model calls a tool to append it to a file.
  • Read. At next startup, the program reads the file contents and places them into context.

4.1 The tool and the startup read

MEMORY = pathlib.Path("memory.md") # cross-session memory lives here def remember(fact): """Append one long-term fact to memory.md.""" with MEMORY.open("a", encoding="utf-8") as f: f.write(f"- {fact}\n") return "Noted." TOOLS = [{"type": "function", "function": { "name": "remember", "description": ("Record a fact that will still matter in later conversations, such as " "long-term preferences or stable habits. Do not record things that only " "matter within this conversation."), "parameters": {"type": "object", "required": ["fact"], "properties": {"fact": {"type": "string", "description": "One sentence stating what it is"}}}, }}] # At every startup, place memory contents after the system prompt memory_text = MEMORY.read_text(encoding="utf-8") if MEMORY.exists() else "(none yet)" SYSTEM = ("You are Xiaoji, my assistant for organizing study notes. Keep answers short.\n\n" f"[Long-term memory about me]\n{memory_text}") messages = [{"role": "system", "content": SYSTEM}] 

Three design notes.

What to record and what not to record is written in the tool description. The model's decision to call remember rests entirely on that sentence. A vague description produces either over-recording or under-recording, and both are hard to debug after the fact.

Memory is appended after the system prompt, inside the static prefix. It does not change during a run, so it does not disturb the cache. Facts recorded during the current run are already in the trajectory and enter the prefix at the next startup.

The read is deliberately crude: load the entire file. With a handful of memory entries this is the simplest correct approach. It stops being correct as the file grows — a limitation taken up in section 6.

4.2 Wiring the loop

The loop is nearly identical to the previous part's, with one added line: before each new question arrives, decide whether compaction is needed.

def run(question, max_steps=10): messages[:] = compact(messages) # check budget before each new question messages.append({"role": "user", "content": question}) for step in range(1, max_steps + 1): resp = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS) msg = resp.choices[0].message messages.append(msg) if not msg.tool_calls: return msg.content for call in msg.tool_calls: result = remember(**json.loads(call.function.arguments)) print(f"[step {step}] {call.function.name} {call.function.arguments}") messages.append({"role": "tool", "tool_call_id": call.id, "content": result}) return "Step budget exhausted before finishing." while True: q = input("me: ").strip() if q in ("exit", "quit", "q"): break print("Xiaoji: " + run(q)) 

Compaction is placed at the moment a new question arrives rather than mid-step. At that point the previous round is complete, so a tool call and its result cannot be split apart by the cut.

Put the three code blocks in one file and the agent is at v0.4. Two runs of it follow.

Run one. State a preference, then converse for several turns.

me: From now on, format my note titles as "date-topic", e.g. 1118-context. [step 1] remember {"fact": "user wants note titles formatted as date-topic, e.g. 1118-context"} Xiaoji: Noted — title format is date-topic, e.g. 1118-context. I'll use it for all notes. ... (three more turns) [compact] folded the earlier 4 messages into one summary 

Exit, restart, ask again.

me: Give today's note about memory a title. Xiaoji: What's today's date? Give me month/day (e.g. 5-6 → 56), and the title will be "56-memory". 

Restarted, the agent still remembers the title format. It also still does not know today's date — the same shortcoming identified in part one.

4.3 The memory was wrong

Opening memory.md revealed something worth dwelling on. The recorded entry was half a sentence longer than what was actually said:

"…date in four digits, without leading zeros."

That instruction was never given. The model inferred it from the example and recorded it as fact. The 56 in the second run was produced by following that wrongly recorded memory.

Memory written by the model will contain errors, and will add statements the user never made. A wrong memory does not fail once; it propagates into every later conversation that reads it. The countermeasure adopted here is simple: store memory as plain text a human can read, open the file periodically, and correct errors by hand. A machine-readable format would be more convenient for the agent and harder for a person to audit — for memory, auditability wins.

5. What to Record and What Not To

Should record Should not record
Long-term preferences: title format is date-topic Requirements tied to one turn: answer more briefly this time
Stable facts: I am learning about agents; I have finished tool calling Transient state: I am currently on the third paragraph
Repeated corrections: do not abbreviate in English Intermediate task results — those belong in an offloaded file
Sensitive data such as passwords and ID numbers

The test is one question: would this still apply tomorrow, on a different task? If yes, record it.

Three problems accumulate as memory is used. None of them is handled in v0.4; they are noted here so the limitation is explicit.

It grows. Every startup loads the whole file, and a large file occupies a large share of the context. Periodic consolidation is required.

It goes stale. A preference from six months ago may no longer hold. Entries should carry a date, and newer entries should override older ones.

It contradicts itself. Append-only storage accumulates both "prefers short" and "prefers detailed". A write should first check for a nearby older entry and amend it rather than adding a second one.

Memory is not hard to store. It is hard to govern.

6. Common Misconceptions

Original assumption What actually happens
The model has a memory feature The model itself does not remember. Memory lives outside and is loaded into context when needed
Staying under the limit is fine Degradation appears well before the limit; set a budget below it
Compression is lossless Summaries drop detail, cost an extra call, and invalidate cache
More memory is better More memory consumes context, goes stale, and contradicts itself
Memory is self-written, so it is reliable It errs and adds statements never made; it needs review and editing

Summary

  • The model has no memory. All memory is context work: either the content is still in the window, or it lives outside and is loaded back when needed.
  • Context degrades before it fills. Set a budget well below the ceiling.
  • A long trajectory has three exits: trim, compress, offload. Compression invalidates cache, so accumulate and cut in large steps rather than a little at a time.
  • Cross-session memory is a file plus two actions: write and read.
  • Record only what still applies tomorrow on a different task. Memory must be readable and correctable by a person.

Version state:

Xiaoji v0.4
Added since last version Automatic compaction when budget is exceeded; one memory file and a remember tool
Can now do Long conversations without overflowing; preferences survive a restart
Still broken Memory and notes are loaded in full; both degrade once there is much of either

The next version has to address memory consolidation and selective loading. For now, the honest assessment is that this agent remembers in the way a notebook remembers — accurately enough to be useful, and only as correct as the last person who checked it.

International access: [https://4sapi.com](https://4sapi.com)
Domestic access: [https://4sapi.org](https://4sapi.org)