I've written 76 posts here, and I can no longer remember which one has the number I'm looking for. So I built a small tool that answers questions from them, with a link to the section each answer came from. Everything model-related runs on DigitalOcean Inference: the embeddings, a reranker and the model that writes the answer. The code is in oceanforge/post-search.
The tool was the easy part. What I wanted to know was how much each piece earns its place, starting with whether embeddings beat plain keyword search on a collection this size.
How it works
The tool downloads every post through the dev.to API and splits each one at its ## headings, which gives 664 sections. Each section is embedded together with its post title and heading.
A question goes through three steps:
- Embed the question and take the 20 nearest sections.
- Reorder those 20 with the
bge-reranker-v2-m3reranker. - Give the top 5 to Claude Haiku 5.5, which answers from them only and cites them by number.
It runs from the command line, and there's a one-page web UI on top for trying questions:
There's no vector database. 664 vectors fit in memory, and a matrix product plus a sort finds the nearest ones in about 16 microseconds on my laptop. Embedding the whole collection took about 180,000 tokens and cost under a cent with either model.
The test
To measure retrieval, I needed questions with known answers. I picked 60 sections at random and had Sonnet 5.5 write one question for each, phrased the way someone who hadn't read the post would ask it, without reusing the section's distinctive words. That second rule matters. A question that copies the post's wording makes keyword search look better than it is.
I added 8 real questions that readers had left in the comments, the ones that make sense on their own. For each question, I checked whether the post it came from was the first result.
Keyword search was nearly as good
Plain BM25, with no model at all, put the right post first for 50 of 60 questions. Both embedding models got 55. On a few hundred sections about distinct topics, most questions share enough words with the right post for keyword search to find it.
The two embedding models tied, even though qwen3-embedding-0.6b costs twice as much per token. The reranker helped most where the first pass was weakest. It lifted BM25 from 50 to 56, nearly level with the best combination, while bge-m3 stayed at 55 with or without it. The best setup, qwen3-embedding-0.6b plus the reranker, got 57 first and all 60 in the top 5.
Combining BM25 with embeddings, which is the usual advice, didn't improve the first result. It did get every right post into the top 5, which is what the answering step needs.
Where the methods differed
The reader questions were where embeddings showed what they're for. One reader asked, under a post about zero-downtime deploys, "what about when I am using LB controller instead, does draining happens there?". BM25 ranked the right post 15th. The question leans on "LB controller", and the post never says "controller" and says "LB" only twice. Every embedding model put it first.
Some "misses" weren't really misses. One generated question asked why pkill -f kills the shell it runs from. I've made that same mistake, and written it up, in four different posts, so a method that returned one of the other three wasn't wrong. It just didn't match my answer key. A collection by one author repeats itself more than I'd assumed.
The answers
With the best retrieval in front of it, Haiku 5.5 cited the post each question came from in 58 of 60 answers. When the sections didn't contain the answer, it said so. One reader asked how long a Redis BGSAVE takes on a realistic dataset, and the post never covers that. The answer said the sections don't mention it and stopped.
The failure worth knowing about came from questions addressed to me. A reader had asked under the Droplet snapshot post, "Did you try one where Postgres had synchronous_commit = off?". The answer began "Yes", and went on to describe a different post, the one comparing Postgres and MongoDB, where the managed Postgres did have synchronous_commit off. Every sentence in it was true. The answer to the reader's actual question was no. A question in the second person, "did you", gets matched to any post where I did something similar.
Answering all 68 questions cost $0.037, about $0.0005 each. Finding the sections took a median of 1.4 seconds, most of it two network calls, the query embedding and the rerank. Writing the answer took a median of 4.2 seconds.
What it cost
All the inference for this project, including every test run, came to about 16 cents. More than half of that was Sonnet 5.5 writing the 60 test questions. The tool itself costs almost nothing to run. Embedding everything again costs under a cent, and a question costs about $0.0005.
What I got wrong
I expected the combined BM25 and embedding search to come out on top. It had the best top 5, but it put the right post first no more often than either embedding model alone.
My first download stopped halfway with a JSON error. dev.to had rate-limited me, and my code tried to read the refusal as JSON. The fetcher now backs off and retries.
The first answer I tried ended in the middle of a word. I raised the token limit and ran the same question again, and it finished normally. I couldn't make it happen again in 68 more answers, so I can't say what caused it.
And the first CI run of the repository failed on my own quoting in the workflow file, before it got as far as the tests.
Caveats
This is one author's 76 posts, each about a fairly distinct topic, which is an easy case for search. The 60 test questions were written by a model from the very sections they're scored against, and some kept a product name like Temporal where no plainer word existed, which helps keyword search. Eight reader questions are too few to rank methods on. Each method ran once.
What I'd take from it
- Try BM25 before embeddings on a small collection. Here it found the right post first five times out of six.
- If you add one model, add a reranker. It closed most of the gap for keyword search, at about three thousandths of a cent per question.
- The cheaper embedding model,
bge-m3, did as well as the one that costs twice as much. - Watch questions that say "you". They ask about a specific post, and similar work in another post will be offered as the answer.
- Under a thousand sections doesn't need a vector database. A NumPy array in memory was enough.



