I built a local-first memory for coding agents. It's a Rust binary and one SQLite file, using MiniLM embeddings plus BM25 keyword search, with no LLM in the retrieval path and no re-ranker.

On LongMemEval S (470 scored questions) it scores R@1 0.913 (429), R@5 0.987 (464) and R@10 0.996 (468).

palace-rs recall on LongMemEval S

I wanted more, mainly at R@1 and recall_all (every gold session in the top K). But all while preserving the strict rules of NO LLM, no re-ranker model and not tuning for the benchmark.

Claude Code did most of the legwork:

  • offline replays of chunk-score dumps
  • a DEV/TEST split by question-id hash
  • a "guard" suite (my coding eval plus probes against a snapshot of my real palace), so a benchmark gain couldn't quietly hurt real use

What we tried

Idea LongMemEval Real-palace guards
Fix two WordPiece tokenizer deviations 0 questions changed probes -1/200
Embed at MiniLM's trained 128 tokens, not 512 R@1 +1, recall_all@5 -7 probes 0.900 -> 0.875, fail
256 tokens +4/-4 at R@1 fail
Re-weight cosine vs BM25 (0.70 to 0.85), DEV only +/- 2 questions probes 0.885 -> 0.790, coding eval 0.922 -> 0.882
Count BM25 only when cosine > 0 <= 1 question probes -4 to -6 pts
RRF, min-max, z-score, saturating BM25 all within 3 questions of baseline n/a
Split two-part questions and merge best scores R@1 +2/-0 DEV, 0/0 TEST unchanged
Treat advice requests as preferences R@1 -5 n/a

The only clean win was 429 -> 431 at R@1. BEAM 100K is already 355/355 at R@5.

A week of tuning: almost everything is noise

How I read it

  1. One question is 0.2%. Most "improvements" were 1 to 4 questions, moving in both directions.
  2. The right sessions are already in the top 30 for 469 of 470 questions. The losses come from ordering, not from failing to find them.
  3. Fusion is saturated: every way of combining cosine and BM25 lands within 3 questions of the others.
  4. Every LongMemEval gain cost my real-palace probes more than it gained.
  5. The leftover misses are reasoning, not retrieval: questions about dates and counting across sessions. The paper's own gains there come from an LLM, which I avoid on purpose.

So my answer is both, but mostly the benchmark. For a pure retriever, S is close to its ceiling. At R@1 I'm at the limit of a small embedding model plus BM25. The next real step is probably a local cross-encoder reranker, not more tuning.

The Claude Code part

The most useful thing it did was keep me honest. It flagged when a "win" was 2 questions, reported failed guards plainly, and stopped when I said "stop if we don't have clear wins". Nothing from the week shipped, and I think that's the right result.

Questions

  • At 98%+, is LongMemEval S still a useful signal for you, or have you moved to M, BEAM 1M or your own evals?
  • Has anyone gotten a real R@1 lift from a local cross-encoder on conversational memory without bad latency?
  • Is there a non-LLM trick for the date questions that I'm missing?

Repo, runner and per-question outputs: https://github.com/AncientiCe/palace-rs