I built Atlas AI, a private copilot where you upload documents, ask questions, and every answer cites the page it came from. The part I cared about most was tenant isolation: one workspace's text must never reach another's answer.
Demo: https://atlasai-beta.vercel.app (continue as guest)
Code: https://github.com/yashwanthvarmakonduru/Atlas-AI
The setup
Every search is filtered to one workspace by server code, never by the model. Search is hybrid: a vector search for meaning and a keyword search for exact names, merged with reciprocal rank fusion. I wrote an isolation test suite and it passed.
The bug
Then I deleted the workspace filter from the keyword search to see what would break. The suite still passed.
A later lookup repeated the filter and quietly dropped the leaked results. Two layers of defence meant the failure of the first one was invisible. My tests were checking the final answer, not each layer.
The fix
I exported each search stage and attacked them one at a time. After that, every deliberate break failed a named test. A test only tells you something once you have watched it fail.
Where the numbers ended up
My first retrieval benchmark scored 71.7% recall@10, which honestly stung. After tuning chunk sizes and a reranker I got to 89.2%, still just under the 90% I'd promised myself. What finally got me past it was swapping the embedder for a small semantic one that runs inside the API, bge-small-en-v1.5. That took it to 95.4%.
One caveat: the 240 questions are generated, so this tells me how search behaves on near identical sentences, not on messy real-world documents. I'd love to test it on those next.
Has a test of yours ever passed when it shouldn't have? Mine did, and I'm curious what yours missed.
