"It learns over time" is the most common claim made about AI agents with memory, and the hardest one to show. You can't just say it. You have to run the same question at different points in the agent's life and put the answers side by side.
That's what I did with DejaVu, an incident-response agent that remembers past incidents and which fixes worked. Same alert, same model, same code; the only thing that changes is how much history is in memory. This article walks through the three answers and how the experiment is built.
Repo: https://github.com/cybersleuth25/Deja-Vu
The question
The alert is the same every time:
Checkout 500s: DB connection pool exhausted (payments-api)
It's a good test case because the "obvious" fix, restarting the connection pool, is the one that keeps failing for this team. An agent that learns should stop recommending it.
0 incidents: the generic answer
With an empty memory, the agent can only fall back to standard advice:
Do first: Restart connection pool (generic); Restart service (generic); Increase pool size (generic)
Avoid: none
No similar incidents in memory.
This is the "before" picture. It's what you'd get from any chatbot, and its first suggestion is precisely the one that has never worked here.
5 incidents: the first match
After the five oldest incidents are in memory, one of them is a DB pool outage:
Matched: INC-0004
Do first: Roll back pool config (1/1)
Avoid: Restart connection pool (failed 1/1)
The change "Tidy db config" touched db/pool.py 3h ago; a change to the same files preceded INC-0004.
One past incident is already enough to flip the advice. The agent now leads with the rollback, warns against the restart, and notices that a recent change to the same file came before the last outage too. With a single data point, "1/1" is a hint rather than proof, and the answer shows the count so the engineer can judge that.
20 incidents: a track record
With the first 20 incidents in memory:
Matched: INC-0004, INC-0009, INC-0014, INC-0020
Do first: Roll back pool config (3/4); Patch connection leak (1/1)
Avoid: Restart connection pool (failed 4/4)
This outage on payments-api matches 4 past incidents... Roll back pool config worked 3 of 4 times; Restart connection pool failed 4 of 4.
[Screenshot 1: terminal output showing the three answers at 0, 5 and 20 incidents]
Now it's a track record. The restart has failed four times out of four. The rollback worked three times out of four, and the one time it didn't, the cause was a connection leak, which is why "Patch connection leak" now appears as a second option. That's the kind of nuance an engineer only gets from someone who was there for all four outages.
Nothing was retrained between these runs. The model is the same at 0, 5 and 20. Only the memory grew.
How the experiment works
The script sorts the incident history by date and replays it in windows: the first 0, then the first 5, then the first 20. For each window it asks the agent the same question:
def run(steps: list[int], live: bool = False, scenario: str = "checkout_500s") -> list[dict]: history = sorted(load_seed(), key=lambda i: i["opened_at"]) alert = detector.primary_alert(sim_events.generate(scenario)) live_mem = HindsightMemory() if live else None rows = [] for n in steps: window = history[:n] if live_mem: mem = WindowedMemory(live_mem, {i["id"] for i in window}) else: mem = MockMemory() mem.retain_many(window) result = analyzer.investigate(alert, window, mem, use_memory=True, llm=None) rows.append({"n": n, **result}) return rows Two details keep the experiment honest:
- Time order. The windows follow the real order of the incidents, so at "5 incidents" the agent only knows what the team would have known at that point.
- The live run uses the real memory. With
--live, recall goes to the actual Hindsight bank. A thin wrapper drops any recalled incident outside the current window, so the agent can't peek at the future even though the bank holds everything.
The LLM is switched off for this run (llm=None), so the text comes from a fixed template. That's on purpose: if the answer changes, it's because the memory changed, not because the model phrased something differently.
What "getting better" means here
Looking at the three answers, the improvement is concrete:
| Memory | Recommends the failing fix first? | Warns against it? | Cites past incidents? |
|---|---|---|---|
| 0 incidents | Yes | No | No |
| 5 incidents | No | Yes (1/1) | 1 |
| 20 incidents | No | Yes (4/4) | 4 |
And it keeps going after that. In the live app, every time an engineer clicks Worked or Failed on an action, the verdict is retained into Hindsight memory, and the next similar alert includes it. The learning curve doesn't stop at 20; it's what happens every time someone records an outcome.
[Screenshot 2: the app's incident view with the Worked and Failed buttons]
If you want the background on why this is different from fine-tuning or plain retrieval, Vectorize's What is agent memory explains it well, and the Hindsight docs cover how retain and recall work.
Limitations
- The history is synthetic and clean. The 32 incidents were written for this project, with patterns planted on purpose. Real histories are messier, so a real curve would rise more slowly.
- It depends on people recording outcomes. If nobody clicks Worked or Failed, there's no track record to learn from.
- One question, one curve. The experiment uses a single alert type. Other incident types improve too (credential stuffing goes from generic advice to "Block ASN at WAF worked 3 of 4"), but I only built the side-by-side for this one.
Takeaway
If you claim your agent learns, show the curve: same question, growing memory, answers side by side. For DejaVu, the jump from "restart the pool" to "restarting the pool failed 4 of 4, roll back the config instead" is the whole product in one table, and it happened without retraining anything.
Hindsight docs: https://hindsight.vectorize.io/
Hindsight on GitHub: https://github.com/vectorize-io/hindsight
What is agent memory: https://vectorize.io/what-is-agent-memory
Tags: @Code.in
