Suppose a policy chatbot says a permit expires after 30 days and cites a policy document. The document now says 45 days, but the chatbot keeps giving the old answer. A prompt change might hide the symptom; it will not tell you where the old text entered the response.
I use a bounded trace for this kind of failure. The example below comes from my own synthetic reproduction, not a client incident. It has two invented documents and an extractive answer function, with no LLM call. That keeps the observations testable without claiming to measure model accuracy.
Capture one failing question
Start with a single question that failed after a known source edit. Save only data you are allowed to inspect:
| Evidence | What it answers |
|---|---|
| The exact question and request time | Did two runs ask the same thing? |
| Old and current source excerpts, document IDs, and edit time | Which version should the answer cite? |
| Ingestion/index version or completion marker | Was the edit available to retrieval? |
| Retrieved chunk IDs, text excerpts, and source versions | Did retrieval return current or old material? |
| Answer-cache key or safe hash, hit/miss, and TTL | Was an earlier answer reused? |
| Final answer and displayed citation | Does the citation support the answer against the current source? |
Redact names, secrets, personal data, and proprietary passages before sharing a trace. A question plus a screenshot of the wrong answer is usually too little to locate the fault.
Locate the first stale boundary
Compare the trace at each handoff, in order:
- Source → ingestion. Did the edited document reach the ingest pipeline, with a new source version?
- Ingestion → retrieval. Does retrieval return a chunk from that version for this question?
- Retrieval → answer cache. If the retrieved chunk is current, did the answer layer reuse an older cached answer anyway?
- Answer → citation display. Does the displayed document ID and excerpt actually appear in the current source?
The first mismatch narrows the next test. An outdated index is a different problem from an answer cache that ignores source changes. A correct answer paired with the wrong citation can be a display or metadata mapping fault. I would not prescribe one fix before seeing that boundary.
In the synthetic reproduction, the initial source says 30 days. After changing it to 45 days, the naive answer cache hits an entry keyed to the question and returns the 30-day text. The cited excerpt is no longer in the current document. The trace shows cache_hit: true and citation_valid: false. A source-aware key misses after the edit and returns the 45-day excerpt; this demonstrates the cache mechanism for this case, not a universal RAG fix.
| Request | Answer | Cache hit? | Citation in current source? |
|---|---|---|---|
| Before the edit | 30 days | No | Yes |
| After the edit, naive cache | 30 days | Yes | No |
| After the edit, source-aware cache | 45 days | No | Yes |
That table matters because the wrong answer and the wrong citation have the same origin in this example: an answer saved against an earlier source. In another system they could be separate failures, so the trace should keep the answer text and the cited excerpt as distinct fields.
Write the acceptance check before the patch
For this one failure, I would agree on an observable result: immediately after the source edit has completed ingestion, the same question must return the 45-day statement, cite text present in the current document, and identify the expected document. A repeat request may hit the current answer cache. The old 30-day citation must not reappear while the previous entry's TTL is still live.
The public unit and HTTP tests show this pattern with a fake clock and cache. They cover the stale cache hit, the fresh post-edit miss, expiry, a missing citation, and an invalid source hash. A separate local Dragonfly smoke test is recorded in the README. These checks establish the behavior of a small demo; they do not prove production retrieval quality, concurrent update safety, or a real customer's outcome.
In a larger system, an answer can depend on retrieved chunks, prompts, model or generator version, and access rules. Cache invalidation must reflect the dependencies that actually produced the answer. Every document-update path needs to advance the source state used by retrieval and citation checking. If concurrent edits are possible, the source update must be atomic or coordinated by an authoritative ingestion process.
The useful deliverable from a one-bug diagnosis is a short evidence chain: where the old text first appears, which hypothesis the trace supports, a reproducible acceptance check, and what remains unverified. That lets a team decide whether the change belongs in ingestion, retrieval, caching, generation, or citation display.
If your existing chatbot has one reproducible stale-answer or citation failure, I offer a USD 149 diagnostic handoff with a pre-purchase scope review. Please share only redacted source excerpts and safe traces. The diagnosis documents the observed failure path and a regression check; code changes are scoped separately.
