What RAG should search, and what an index should answer

This is part 2 of the series "Adopting RAG — Grow your AI on your own accumulated judgment."

Series contents

1. Adding RAG: store your own judgment and grow your AI
2. What RAG should search, and what an index should answer (this article)

0.726 and 0.713. Those are the similarity scores our RAG gave two architecture decision records for the same question: the current rule for passing work between repositories, and the older rule it partly replaced. A gap of 0.013. Nothing in either number says which one is still in force.

It gets worse when the question uses the old decision's own words. In our example, a query worded after a replaced record's title got exactly one design record handed to the AI: the replaced one. The three current records on the same topic do not make the list, and the injected line carries no hint that the record is out of date.

This is part 2 of a series. Part 1 set up the RAG: every time I send a message in Claude Code, a hook searches my notes, past conversations and our shared rules by meaning, and hands the closest few to the AI before it answers. Here is how we stopped it from passing replaced decisions off as current ones.

The short version

• Similarity finds what is related, not what is valid. A replaced decision can score almost as high as the decision that replaced it.
• The index answers "which one is current?" It is the list of ADRs we rebuild from one fixed rule on every call, covered in an earlier article (Too many ADRs made our AI follow outdated decisions).
• The RAG keeps searching, and stamps each ADR with the index's answer. A replaced record arrives as "⚠️superseded → successor" instead of looking current.

Two scores you cannot tell apart

We used to hand off work between repositories on an Asana board. We have since moved it to usesources, one of our own services. The new handoff record documents the move; the old board record is the one it partly replaced. Both sit near the top, a hair apart.

A similarity score is a measure of how close two pieces of text are in meaning. Whether a record has been replaced was never an input. Ranking alone has nothing to separate old from new.

The same record, before and after the mark

We keep every design decision as an ADR, and never delete one when a decision changes; the new record declares what it replaces. Our in-house Claude Code plugin suite, ELN workflow, has 191 of them. 18 are not current (superseded, proposed or rejected) and 8 are partly superseded. The RAG searches all of them.

To show the problem plainly, the query below is worded after the title of a superseded Amplify record, and the same selected items were run through the code from just before the fix and the code after it.

Before: a plain line that reads like policy. The Amplify record was replaced by a rule in another repository (builds run on GitHub self-hosted runners). An AI that judged from this line and its description alone, without opening the record, could treat a decision we no longer use as current. (Opening the body would trigger the read-time warning from the earlier article.) The header does say "reference, not instructions", but a replaced ADR is more dangerous than an old note: it can flatly contradict the decision that came after it.

After: the same line starts with "⚠️superseded → external". "External" means the successor lives in another repository.

Every status the index can return maps to one mark:

Why it is built this way

Why not just drop replaced records?

A question about the old decision itself should still find it, and the mark names the successor, so the AI has a path forward instead of a gap. I chose to mark rather than hide.

Why not a quick "is this superseded?" check inside the RAG?

Because it would be a second rule for a question the index already answers, and two rules drift. The RAG calls the index's own parser. When the index's rule changes, the marks change with it, and a test checks that both agree.

Why read every file to judge one?

A record does not know it has been replaced. That is decided when some other record says "this replaces that one". So the RAG loads the whole folder and caches the result under a hash of every file's name and contents. Not modification times: a copy or restore can keep the old timestamp, and flipping "Accepted" to "Rejected" leaves even the file size unchanged. Before shipping, a cold load was measured to fit inside the hook's two-second limit.

What if the line does not fit?

An injection is capped at 2,048 bytes. Descriptions shrink first, down to nothing. The mark never shrinks. If a line still does not fit, the whole line goes, because a replaced record without its mark looks current.

What does not get a status mark

Only design records get a mark with their status from the index.

This is what the AI receives for the Asana query. The first line is a memory note from when we shipped the old board. It has no status mark, and that is deliberate. A memory note records what happened at the time. It has no "replaced by" field, so the only way to give it a status mark would be for the RAG to guess whether it is stale: the second rule again. Past conversations are treated the same way. An old note in the injection is a reminder, not the current rule.

Asking the index directly

Ask the index about Amplify builds and one command shows four records: three live, one superseded. Opening the superseded one names its replacement. None of that was visible from the single line the RAG handed over. Absent from the injection does not mean absent from the repository.

In practice, the AI gets a few related records with every message. Any ADRs among them carry their status; a mark sends it to the successor, and when it needs the current rule it runs the index command.

Wrapping up

Similarity is how the AI remembers a decision you made months ago, even when you describe it differently today. It is not how the AI should decide whether that decision still stands. Keep a store of your own judgment long enough and a growing share of it will be judgment you have since replaced, so give that question to something that can answer it exactly.

Terms

• RAG: retrieval-augmented generation. Before the AI answers, something searches your own records and hands it the closest ones. In this series it is the feature we call memory recall.
• ADR: architecture decision record, one document per design decision with its reasons. Each starts with a status (accepted, amended, proposed, superseded, rejected) and fields for what it replaces and what replaced it.
• The index: the list of which ADRs are current and what replaced what, rebuilt from those fields by one fixed rule on every call, queried with the eln-adr command. It is not the embedding index from part 1.

About the figures

All four figures are real output of the RAG command (recall.py) and the index command (eln-adr) run in our repository, called by hand rather than through the hook, and laid out as a terminal with the text unchanged except as listed in the next section. The grey lines starting with # and the command lines starting with $ are captions we added; they are not tool output. The query wording was chosen for this article to match the titles of the replaced records. The "before" half of the first figure is the same selected items rendered by the version of the code from just before the fix.

What the figures leave out or replace

• The tools print Japanese only, so in these English figures the Japanese text, including the queries, is replaced with English translations; record numbers, scores, file names (apart from the masking below) and line order are unchanged.
• Command paths are shortened.
• An internal number in the injection header is removed.
• Internal repository names are replaced with <plugin> and <in-house standards>, and conversation IDs with <conversation>.
• Titles and descriptions that name internal mechanisms (conversation titles, design record titles, descriptions and file names, memory names and descriptions, and script and hook names) are masked with ■.

In this series, I show step by step how I store my own decisions and rules as records, and use RAG so the AI draws on them every time, with our own implementation and measurements.
If this was useful, please like and follow.

Our products are listed at https://www.eln.ne.jp/products

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.