Production Problems in RAG and How to Solve Them

Every RAG tutorial ends at the same place: chunk the documents, embed the chunks, insert the vectors, ask a question. That is day one. On day two someone edits a document, and nothing in the tutorial tells you what to do. This post walks through the nine problems that show up after that, in the order you usually hit them. Each fix solves one problem and exposes the next. You can replay all nine in a browser lab I added to my Advanced Local RAG demo.

Why this is the hard part

Building a retrieval pipeline is a weekend project. Running one is not, because the corpus is never finished. Policies change, a typo gets fixed, a product gets renamed. Every one of those edits has to reach the vector index correctly, cheaply and without users noticing.

Most of the failures below share one dangerous trait: nothing errors. There is no exception and no alert. The index is just quietly wrong, and retrieval returns a confident, well-formatted answer built from the wrong text.

The lab is a new page in the existing demo. It runs entirely in your tab: no server, no API key, no model download. Every number in this post is computed live on page load (about one second) and can be replayed from a URL like production.html?rung=4.

The test bed

It is a simulation of the mechanics, not a benchmark of any product. The point is that every failure is real inside the simulation: the crash really leaves mixed versions, the model swap really breaks similarity, the dropped event really leaves a stale vector.

Problem 1: One edit re-embeds the whole corpus

The problem. The first indexer is a loop: for every document, chunk, embed, insert. When something changes, the simplest move is to run the loop again. At a few thousand vectors nobody notices. At twenty million vectors and 2,000 embeddings per second, that is 2.8 hours of embedding for one edit.

The fix: store a document ID on every chunk. When a document changes, delete that document's vectors with a metadata filter and re-embed only that document.

In the lab, one sentence added to the handbook costs 258 embeddings with a full rebuild and 18 with document IDs, 93% fewer. More importantly, the cost now scales with the size of one document, not with the corpus.

upsert({ id, vector, metadata: { docId: 'handbook' } }) // on change: replace only that document's vectors await index.delete({ filter: { docId } }) for (const c of chunk(latest)) await upsert(embed(c), { docId }) 

But then: long documents are hundreds of chunks. One added sentence still re-embeds all of them.

Problem 2: A one-sentence edit re-embeds the whole document

The problem. Deleting and re-embedding a whole document is wasteful when one sentence changed.

The fix: content-hash every chunk. Re-chunk the new version, compare each chunk's hash with the hashes already in the index, and embed only the new ones.

This is the step where the lab gets interesting. With fixed 400-character windows, the edit lands in chunk 3. Chunks 0 to 2 are reused. Then 15 of 18 hashes change anyway. Hashing saved three embeddings.

But then: why did chunks that contain unchanged text get new hashes?

Problem 3: The edit shifts every later chunk boundary

The problem. Fixed-size windows are positional. Insert 83 characters early in a document and every later window starts 83 characters further on. The text in each window barely changed, but every byte offset did, so every hash after the edit is new.

The fix: chunk along boundaries that survive edits. Cut at headings, sections and paragraphs, and prefix each chunk with its heading path. Now an edit inside one paragraph changes exactly one chunk.

Same edit, same document: 1 of 60 chunks re-embedded, 59 untouched. Structure-anchored chunks are smaller and more numerous, but a local edit stays local. (In my previous post on the Chunking Lab, structure-aware chunking also won on retrieval quality, so this fix pays twice.)

for (const s of doc.sections) for (const p of s.paragraphs) chunks.push({ text: `${doc.title} › ${s.title}\n${p}`, section: s.id }) // long paragraphs: split on sentences, never on offsets 

But then: updates are now small and fast, until one crashes halfway through.

Problem 4: A crash leaves two versions mixed together

The problem. Version 13 of the handbook raises annual leave from 15 to 20 days and remote work from 2 to 3 days per week. The change touches four paragraphs in four sections. The indexer updates in place, one paragraph at a time, and crashes after two.

Nothing fails at query time. Ask "how many days of paid annual leave per year" and retrieval returns section 3 from v13 (20 days) and section 8 from v12 (15 days), with similar scores. The index contradicts itself, and the LLM will pick one.

The fix: write each update as a new document version. Writers never touch the live version. They write v13 next to v12, and readers filter on the active version only.

With versioning, the same crash leaves 30 of 60 v13 chunks written and v12 still active. Every reader sees one consistent answer (15 days), and the v13 build resumes on retry. An older answer that is consistent beats a newer answer that is half true.

for (const c of chunks) await upsert(embed(c), { docId, version: 13, hash: c.hash }) search(q, { filter: { docId, version: versions.active(docId) } }) 

But then: versioning protects you from crashes, but only if you don't activate the new version too early.

Problem 5: Search during a deploy returns half a document

The problem. Most vector databases acknowledge an upsert before it is searchable. Writes are queued, batched and indexed asynchronously, often out of order. If the indexer marks v13 active as soon as it has sent the writes, anyone who searches during the deploy reads a version that is only partly there.

The lab searches when 45% of the queue has drained: 27 of 60 v13 chunks are visible. The coverage map shows whole sections missing. The question "how many days per week can I work from home" gets home-office equipment and a gym-access FAQ, because both remote-work paragraphs have not landed yet.

The fix: an atomic version switch. Build the complete new version first. Wait until the index confirms every write. Then flip the active-version pointer in one write. Readers stay on the complete v12 until v13 is complete too.

await upsertAll(v13Chunks) // queued, not yet visible await waitUntil(() => index.count({ docId, version: 13 }) === v13Chunks.length) await versions.activate(docId, 13) // one atomic pointer write 

But then: every edit now leaves a complete old version behind, and nobody deletes them.

Problem 6: Old versions pile up in the index

The problem. With versioning, every update writes a full new version of the document. Nobody removes the old ones. They cost storage, they slow down every filtered search, and they are waiting for the first bug that forgets the version filter.

After 300 small edits to the handbook, the index holds 18,540 vectors, and 97% of them are stale. Every filtered search scans about 31 times more rows than it needs to.

The fix: garbage-collect superseded versions. After a successful switch, queue an asynchronous, idempotent clean-up job. Keep one previous version for rollback. With GC the index stays at 600 vectors, and the only stale rows are the rollback copy.

But then: the index is lean and consistent. Then the team upgrades the embedding model.

Problem 7: An embedding model upgrade mixes two vector spaces

The problem. A new embedding model is not a better version of the old one. It is a different vector space. Cosine similarity between a v2 query vector and a v1 document vector is noise.

In the lab, the pipeline switches to embed-v2. Over the next days, 25 documents are edited and re-embedded with v2. The other 96 keep their v1 vectors, and queries are now embedded with v2. Recall@5 falls from 99% to 24%: queries can only find the documents that happen to have been edited since the upgrade. Rolling back the query model does not save you either: v1 queries reach only 80%, because the edited documents moved to v2. And again, no error is raised.

The fix: track the embedding model on every vector, and re-index on purpose. Serving is pinned to one model. The new model's index is built in the background as a shadow index, with edits dual-written. Cut over when the shadow index is complete, then delete the old vectors. In the lab, recall holds at 99% during the migration and 99% after the cut-over, at the cost of 540 vectors re-embedded once, deliberately.

upsert(vec, { docId, version, model: 'embed-v2', dim: 1024 }) const m = active.model search(embedWith(m, q), { filter: { model: m } }) await reindexAll({ model: 'embed-v2' }) // shadow index, dual writes await setActiveModel('embed-v2') // cut over when complete 

But then: every layer is now correct, provided it hears about every change.

Problem 8: The update event never reaches the indexer

The problem. Changes reach the indexer as events, and events get lost. A broker restarts, a consumer throws on a malformed payload, a deploy drops an in-memory queue. In the lab, 2 of 40 update events disappear. One of them is the handbook change that moves the laptop refresh cycle from 3 years to 4.

Ask "how often are laptops refreshed" and the answer is every 3 years. The source says 4. Nothing is broken, so nothing alerts.

The fix has two parts.

1. Transactional outbox. The update event is written in the same database transaction as the document, and a relay publishes it with retries. A document can no longer change without its event existing.
2. Reconciliation sweep. On a schedule, compare a content hash of every source document with the source hash recorded in the index. Anything that differs is re-indexed. This catches the failures the outbox cannot, such as indexer bugs, manual deletes and restored backups.

In the lab, the sweep compares 121 source hashes with the index, finds the 2 drifted documents and repairs them. The answer becomes "every 4 years".

await db.tx(async t => { await t.saveDoc(doc) await t.outbox.add({ docId: doc.id, hash: doc.hash }) }) for (const d of await db.docs()) if (d.hash !== await index.sourceHash(d.id)) await reindex(d.id) 

But then: the index is correct now. The answer users see may still not be.

Problem 9: The answer cache serves yesterday's answer

The problem. Popular questions are cached, keyed by the normalised question. The handbook is updated and the index picks up the change correctly, but the cache keeps serving the old generated answer until its TTL expires. The retrieval layer is right. The user still gets yesterday's policy.

The fix: store source versions with every cached answer. A cache entry remembers which document versions it was generated from. On lookup, if any of those documents has a different active version, the entry is invalidated and the answer is regenerated.

cache.set(key, { answer, sources: { handbook: 13, 'kb-12-faq': 4 } }) const hit = cache.get(key) const fresh = hit && Object.entries(hit.sources) .every(([doc, v]) => versions.active(doc) === v) if (fresh) return hit.answer 

Chaos week: all nine fixes together

The second tab of the lab replays one scripted week of production against whichever fixes you switch on: 100 edits on Monday, a crash on Tuesday, a search during a deploy on Wednesday, 300 accumulated edits on Thursday, a model upgrade on Friday, lost events on Saturday and a repeated question on Sunday. Each incident links back to the problem that explains it.

With no fixes, the system handles 1 of 7 incidents, and only because it has no versions to collect. That is the system most tutorials ship.

Turn on the first four fixes (document IDs, hashing, stable chunking, versioning) and the week costs 100 embeddings instead of 25,800, and the crash is harmless. That is the system most teams build after their first incident. It still fails on deploys, bloat, the model upgrade, lost events and the cache.

Turn on all nine and the grade becomes an A.

The fixes are toggled with dependencies enforced: the atomic switch and garbage collection need versioning, and stable chunking only pays off with hashing.

The checklist

None of these fixes is exotic. Every one is a standard data-engineering pattern: idempotent upserts, content addressing, blue/green deploys, compaction, schema versioning, the outbox pattern and cache invalidation. RAG doesn't need new ideas here. It needs teams to accept that a vector index is a derived dataset, and that derived datasets need the same care as any other replica of production data.

Try it

• Live lab: vishalmysore.github.io/advancedRagDemo/production.html
• Jump to a problem: append ?rung=1 to ?rung=9
• Chaos week with a configuration: ?tab=chaos&fixes=docIds,hashing,stable,versioning
• Source: github.com/vishalmysore/advancedRagDemo. The simulation is src/utils/prodLab.js, under 400 lines with no dependencies.

The same repository has the browser-native RAG pipeline (hybrid BM25 plus vector search, reranking, WebLLM generation) and the Chunking Lab, so you can follow a document from chunking to retrieval to staying correct in production.

If your team has hit a tenth problem that belongs on this list, open an issue. I'll add it to the lab.