I build Calla, an AI companion app whose whole pitch is memory: you tell it about your week, close the app, and the next day it still knows your name, your job and the people in your life. For about four months, the part of the system that finds the right memory at the right moment was returning the least relevant memories first. Nobody noticed, including me. Here is how that happens, and what I'd tell anyone building long-term memory for an LLM.

Memory is three problems, not one

"The bot remembers what I said ten minutes ago" says almost nothing. That's the context window: a rolling buffer the model can see. Close the app and many products start the next session with an empty one.

Real long-term memory is three separate jobs, and each can fail quietly:

  1. Keeping. After a conversation, something reads it back and decides what was worth remembering. We store short episode summaries ("they're nervous about Thursday's interview; their manager has been difficult since the reorg") plus a few structured facts that changed.
  2. Finding. When a new message arrives, the system searches everything it kept and pulls back the handful that matter right now. That's usually vector search over embeddings, so "my sister" can find the episode about the wedding speech even if you never say "wedding".
  3. Bringing it up. Knowing your sister's name is different from asking, the day after, how the speech went. That needs open loops tracked until they've actually happened.

Our bug lived in step 2.

How an inverted search hides

We used an open-source memory layer for the episode store. Its search returned a score with each hit, and our code treated that score as similarity: higher is better, keep the top few. The store was actually returning a distance: lower is better. So every lookup sorted the memories backwards and kept the least related ones.

Why didn't anything look broken?

  • Every result was still a real memory about the right person. The companion never said anything false; it just pulled up something about a different week. That reads as "a bit vague", not "broken".
  • The fact store hid it. Names, jobs and pets live in a separate structured profile that goes into every prompt. The obvious tests ("what's my name?") passed every time.
  • Logs looked healthy. Searches returned results, latency was fine, no exceptions.

It took a direct test to find it: store one very specific fact, ask about it later in different words, and print what the search actually returned, not what the companion said. The top hit was unrelated; the right memory sat near the bottom.

The fix was small. We stopped going through the library's search and wrote the SQL ourselves (pgvector, cosine distance, ORDER BY distance ASC, with the ordering asserted in a test that uses a known-near and a known-far memory). We also moved recall off the hot path: the next turn's candidate memories are prepared while the current reply is being written, which took memory lookup from about 1.4 seconds per message to under 0.1.

What I'd tell anyone building this

  • Test the judgment, not the plumbing. "Search returns results in 40 ms" is plumbing. "Search returns the right result" is the feature. Build the second test first.
  • Check the sign of every score you didn't write. Similarity and distance look identical in a log line.
  • Keep the fact store and the episode store testable separately. A good profile can mask a broken episode search indefinitely.
  • Use a fixed test the user could run. Ours: mention something with a date attached, close the app, come back the day after, and see whether it knows and whether it asks.

That last test is also how I'd tell anyone to choose an AI companion. Ignore the marketing (mine included) and run it for a week. Knowing is now fairly common. Asking at the right moment, unprompted, is the hard part, and it's still where this whole category, us included, has the most work to do.

I'm Christian, and I build Calla, an adults-only AI companion with memory on the free plan (50 messages a day; Calla+ is $14.99/month). This post was written with help from an AI assistant and checked by me against our code and commit history.