I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests

Ok, I made a thing! LLMs have their... quirks. Namely re-reading the same notes, wandering back and forth, and spending 10x more tokens than needed because I can't be explicit.

So I built Haystack — a semantic router for a markdown vault. Instead of the agent digging through files, it queries: hybrid retrieval (embeddings + corpus-IDF term stats, RRF-fused, cross-encoder reranking via llama.cpp), all local, zero cloud, one 35 MB model, ~10 ms per query. MIT, one-command setup: github.com/diogocccarvalho/haystack.

The part I don't see elsewhere

Your corrections become regression tests. Every retrieval miss you fix becomes a permanent rule AND a QA pair. The audit reindexes and runs every pair — it exits 1 on a regression, so the same question can't silently fail twice.

And the learned rules only pin when they actually beat retrieval: a stored phrasing must outrank the corpus' own best hit for the same query. Scale-free, not absolute-threshold — so swapping embedders doesn't break your corrections. I A/B'd models against frozen QA sets and rejected two (a reranker that was 2x faster but -16pp Recall@3; a 97M embedder at -7pp). Numbers in docs/RESEARCH.md.

The numbers, honestly

The foreign-corpus row matters more than the home-vault row: the chunker generalizes beyond the corpus it was tuned on. Also: llama-server does both the embeddings (bge-small-en GGUF) and the reranking (--reranking flag on ms-marco MiniLM-L12) — one binary, tested with llama.cpp v1.0.0.

The honest part

The agent that uses Haystack daily also built most of it with me. Every gate in the repo was run by the agent against its own index — that's how the setup-script bugs got caught. And docs/SHORTCOMINGS.md is the failure catalog: ten real failures (a reranker overriding user corrections live; a rebuild that once deleted every correction; an over-context chunk killing a whole batch), each one now a design rule.

Try it

./setup.sh --start # model + config + embeddings server, one command python3 haystack.py reindex python3 haystack.py query "that thing about the invoices" 

Any OpenAI-compatible endpoint works instead (Ollama, LM Studio, remote API). Python 3.11+, stdlib + numpy only — no framework, no vector DB daemon, no cloud.

The open question I'd genuinely love answers to: this was tuned entirely on my vault. I want to know how the default config holds up on other people's notes. If you run it, tell me what breaks.