Ask a language model for "similar past investments" when it has never made a single one, and it will happily give you three. Names, amounts, lessons learned, all delivered with total confidence.

That one behavior changed how we built our project. We were creating an agent that audits startup pitches and code, and we realized the scariest bug was not a wrong answer. It was a wrong memory. An agent that invents its own history is worse than one with no history at all, because you will believe it.

So we built an agent where memory is not a feature. It is the rulebook.

Memory is not search

Most "agent memory" today is really search. Save old chats, find the closest ones, paste them into the prompt. That helps, but it is retrieval, and an agent that only retrieves never gets any better.

Real memory does three things: it retains what happened, recalls the right pieces at the right time, and reflects on everything accumulated to form new understanding. Storing is trivial. Turning experience into knowledge the agent can act on is the hard part.

We used Hindsight, an open-source memory system from Vectorize that is built around exactly those three verbs:

python
client.retain(bank_id="my-bank", content="Alice works at Google")
client.recall(bank_id="my-bank", query="What does Alice do?")
client.reflect(bank_id="my-bank", query="Tell me about Alice")

Memories live in banks, recall mixes several search strategies (meaning, keywords, relationships and time), and reflect turns stored facts into new observations. The Vectorize explainer on what agent memory is is a good primer if this is new to you.

What we built

One Python project, two halves, one shared memory bank:

The auditor.The auditor. Paste in a startup pitch or a block of code and get scores, red flags and a verdict (for pitches) or a security score, a bug list and a corrected version (for code). The key difference is the first step: before the AI model says anything, the app searches memory for similar past pitches or bugs. Feedback comes from the project's own track record, not from generic advice.

The self-healing supervisor. It runs a service, watches it crash, and decides whether it is allowed to fix it. A dashboard can start and stop it and shows a live feed of what it did.

Memory as a permission slip

Plenty of "self-healing" demos just let the model patch whatever broke. That is a gamble, so we gave memory the power to say no, using two gates:

• The read gate. The supervisor only tries a fix if memory contains a similar bug that was solved before. No precedent means no edit. It logs the crash, stops, and flags that a human is needed.

• The write gate. A fix is only saved to memory after the patched service survives a set stability window. If it crashes again, the file is restored from backup and nothing is learned.
  The write gate is the one that surprises people. Without it, the agent would store its own failed fixes and trust them later. It would be teaching itself its own mistakes.

In practice this means a divide-by-zero crash in a nightly job can match an earlier "guard against empty input" fix, get patched, and be saved once it proves stable. The next similar crash then has two precedents. A crash with no match gets left alone.

The bugs that taught us the most

Too much recall. Hindsight breaks stored records into smaller facts, and many overlap. Our first lookups returned a huge pile of near-duplicates, which blew past the model provider's request limits and mostly repeated itself. The fix was unglamorous: remove duplicates, shorten each entry, cap the count.

Async collisions.The memory client uses async code internally, which clashed with the dashboard's own event loop in two different ways. What finally worked was giving every memory call its own short-lived thread with its own loop.

Fabricated history. This was the big one. We stopped trusting the model to say what the past looked like. If recall comes back empty, our code overwrites the answer to say no similar case was found. The model never gets the chance to improvise.

Opinions disguised as events. Every audit is saved back to memory, so a few days later an old pitch evaluation could be recalled and described as if real money had been invested. Now every saved audit is labelled as an evaluated pitch, not an actual investment, and the model is told never to present those records as profit or loss.

What we'd tell you before you build this

1. Retention is not retrieval. What you store and what you get back are different shapes. Budget time for cleaning recall results.
2. Gate autonomy on memory, and gate memory on verification. Precedent decides whether the agent acts. Verified outcomes decide whether it learns.
3. Never let the model be the source of "the past". History should come from recall, and your code should enforce that when recall is empty.
4. Label what you write back. An audit, an observation and a fact all look alike in a vector store unless you mark them.
5. Don't pin yourself to one model. One of the models we started with was retired by the provider partway through. A fallback list turned that into a log line instead of an outage.

The honest limits

The fixes are still written by a model, so for anything that moves money you should run the project's tests before restarting. The supervisor edits one file at a time. And it stops after an unfixable crash instead of hiding it in a restart loop, which we prefer, but it is a design choice.

Want to try the memory layer yourself? Start with the Hindsight repository and its documentation. And if you copy one thing from this project, skip the dashboard. Copy the two gates.