Tracing RAG Failures Across Retrieval, Reranking, and Generation

A RAG application can return an incorrect answer even when its vector database, reranker, and language model are functioning normally.

The problem is that each component optimizes a different objective. Retrieval identifies potentially relevant documents, reranking prioritizes candidates, and generation produces an answer from the selected context. A failure at any stage can affect the final response.

The most reliable debugging approach is to trace each stage independently while preserving document identities, rankings, scores, model configurations, and the context actually passed to the LLM.

With structured traces and stage-specific evaluations, engineers can distinguish missing evidence from incorrect ranking and generation errors.

This article explains how to implement that tracing, reproduce common failures, and build an observability workflow suitable for production RAG systems.

1. Why RAG Failures Are Difficult to Diagnose

Consider an enterprise assistant answering questions about internal infrastructure policies.

A user asks:

What is the approved process for rotating production database credentials?
The application retrieves ten documents, reranks them, selects the top four, and sends those passages to an LLM.

The generated answer recommends rotating credentials manually through the database administration console.

However, the organization's current policy requires using an automated secrets-management workflow.

The response is incorrect, but where did the failure occur?
There are three possibilities.

Retrieval failure: The correct policy never appeared among the retrieved candidates.

Reranking failure: The correct policy was retrieved but ranked below less relevant documents and excluded from the final context.

Generation failure: The correct policy reached the LLM, but the model ignored, misinterpreted, or contradicted it.

These failures require different solutions.

Increasing the retrieval candidate count may help the first problem but cannot guarantee that the reranker selects the correct evidence.

Changing the reranker may address the second problem but will not fix a generation model that disregards explicit instructions.

Prompt engineering may help the third problem, but it cannot recover evidence that was never retrieved.

Without stage-level observability, engineers often change multiple components simultaneously and lose the ability to identify which modification improved or degraded the system.

2. Understanding the Three Stages of a RAG Pipeline

A common production RAG workflow looks like this:

User Question | v Query Processing | v Document Retrieval | | Top 20 candidates v Reranking | | Top 5 passages v Context Assembly | v LLM Generation | v Answer + Citations 

Each stage has its own inputs, outputs, configuration, and potential failure modes.

Retrieval: Finding potentially relevant evidence

Retrieval searches a document collection using techniques such as dense vector similarity, keyword search, or hybrid retrieval.

Its primary responsibility is to produce a candidate set containing information that could answer the question.

A retrieval system may fail because of:

• Poor document chunking.
• Incorrect query embeddings.
• Missing or outdated indexed content.
• Inappropriate metadata filters.
• Weak keyword matching.
• Insufficient candidate depth.

A document can be highly relevant to the user's question but absent from the candidate set. Once that happens, downstream reranking cannot recover it unless another retrieval operation is performed.

Reranking: Selecting the strongest candidates

Reranking evaluates retrieved candidates and attempts to order them by relevance to the query.

For example, a cross-encoder can jointly process a query and candidate passage to estimate their relevance.

Reranking may fail when:

• Relevant passages receive unexpectedly low scores.
• Candidate text is truncated.
• The reranker receives a different query from the retriever.
• Important passages are excluded by the final top-k cutoff.
• Model behavior differs from the target domain.

Microsoft's RAG architecture guidance distinguishes broad candidate retrieval from reranking, which aims to improve the precision of the context selected for generation.

Reference: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-information-retrieval

Generation: Producing an answer from evidence

The generation stage combines retrieved context with system instructions, the user's question, and potentially conversation history.

Failures include:

• Unsupported claims.
• Incorrect interpretation of evidence.
• Missing citations.
• Contradictions between the answer and context.
• Excessive reliance on model knowledge.
• Failure to acknowledge insufficient evidence.

An LLM can produce a confident answer even when the supplied context does not support it.

This makes generation evaluation fundamentally different from retrieval evaluation.

3. Why Request-Level Logging Is Not Enough

A typical application log might contain:

{
"request_id": "req_1042",
"status": "success",
"latency_ms": 1840,
"retrieved_documents": 20,
"response_generated": true
}

This tells us that the request completed. It does not explain whether the correct document was retrieved, how reranking changed its position, or what evidence reached the LLM.

For useful debugging, a trace must preserve the relationship between stages.

A more informative trace structure is:

rag.request | +-- query.process | +-- retrieval.search | candidate_count: 20 | index_version: index_v12 | +-- reranking.rank | input_count: 20 | output_count: 5 | model: reranker_v2 | +-- context.assemble | selected_chunks: 5 | +-- generation.invoke model: configured_llm output_tokens: 320 

Each operation becomes a span within the same trace. A trace identifier connects the operations belonging to one request, while individual span identifiers represent the separate processing steps.

This allows engineers to inspect how information moves through the complete pipeline.

4. What Should You Capture in Each RAG Trace?

A useful trace should capture enough information to reconstruct decisions without unnecessarily exposing sensitive enterprise data.

Retrieval span

Capture:

• Query identifier or protected query reference.
• Retrieval strategy.
• Index and embedding model versions.
• Requested and returned candidate counts.
• Candidate document and chunk identifiers.
• Candidate rankings and scores.
• Applied filters.
• Retrieval latency.

Reranking span

Capture:

• Reranker model and configuration.
• Input candidate identifiers.
• Original retrieval ranks.
• New reranking positions.
• Reranker scores.
• Selected candidate identifiers.
• Reranking latency.

Generation span

Capture:

• Generation model identifier.
• Prompt template version.
• Selected context identifiers.
• Input and output token counts when available.
• Generation parameters.
• Response identifier.
• Latency and error status.
• Evaluation results associated with the answer.

Avoid recording raw prompts, confidential document passages, or generated responses by default.

5. Implementing RAG Tracing With OpenTelemetry

OpenTelemetry provides APIs for collecting traces, metrics, and logs across distributed systems.

It can be used to instrument RAG stages independently of the chosen vector database or LLM provider.

For a minimal Python demonstration, install the tracing packages:

pip install opentelemetry-api opentelemetry-sdk 

The following example uses a console exporter so traces can be inspected locally without configuring an external observability backend.

from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import ( BatchSpanProcessor, ConsoleSpanExporter, ) provider = TracerProvider() provider.add_span_processor( BatchSpanProcessor( ConsoleSpanExporter() ) ) trace.set_tracer_provider(provider) tracer = trace.get_tracer("rag.pipeline") 

This creates a tracer capable of recording nested operations. Next, define a small RAG pipeline with three stages.

def retrieve(query): with tracer.start_as_current_span( "rag.retrieval" ) as span: documents = [ { "id": "doc_1", "text": "Credential rotation policy" }, { "id": "doc_2", "text": "Database access guidelines" } ] span.set_attribute( "rag.retrieval.candidate_count", len(documents) ) return documents def rerank(query, documents): with tracer.start_as_current_span( "rag.reranking" ) as span: ranked = sorted( documents, key=lambda item: item["id"] ) span.set_attribute( "rag.reranking.input_count", len(documents) ) span.set_attribute( "rag.reranking.output_count", len(ranked) ) return ranked def generate(query, documents): with tracer.start_as_current_span( "rag.generation" ) as span: span.set_attribute( "rag.generation.context_count", len(documents) ) return "Demonstration response" def run_rag(query): with tracer.start_as_current_span( "rag.request" ): documents = retrieve(query) ranked = rerank( query, documents ) return generate( query, ranked ) print( run_rag( "How are database credentials rotated?" ) ) provider.force_flush() provider.shutdown() 

This example demonstrates nested tracing rather than actual semantic retrieval, learned reranking, or LLM inference.

The document list, sorting operation, and generated response are intentionally deterministic placeholders.

It provides a starting point for instrumenting real components without depending on a particular AI framework.

6. Tracing Retrieval Failures

Retrieval debugging starts with a simple question:

Did the correct evidence appear in the candidate set?

Suppose the expected document is security_policy_v3.

The retrieval output contains:

Rank Document
1 security_policy_v1
2 database_manual
3 access_guidelines
4 security_faq
5 credential_overview

The current policy is missing. This suggests that the problem occurred before reranking. Possible causes include stale indexing, weak embeddings, missing keywords, or overly restrictive metadata filters.

Measure Recall@k

Recall@k measures how many known relevant items appear among the first k retrieved results.

For a query with one known relevant document, the calculation is straightforward.

If the correct document appears within the first ten candidates, Recall@10 is 1 for that query.

Otherwise, it is 0.

For multiple relevant documents:

def recall_at_k( retrieved_ids, relevant_ids, k ): if k <= 0: raise ValueError("k must be positive") relevant = set(relevant_ids) if not relevant: raise ValueError( "Relevant document set cannot be empty" ) retrieved = set(retrieved_ids[:k]) return len( retrieved.intersection(relevant) ) / len(relevant) 

Example:

retrieved = [ "doc_7", "doc_3", "doc_9", "doc_2" ] relevant = { "doc_3", "doc_2" } score = recall_at_k( retrieved, relevant, k=3 ) assert score == 0.5 

This example checks document-level recall. For chunk-based RAG, evaluation may need to operate at the chunk or evidence-passage level.

A single document can contain both relevant and irrelevant chunks, so document-level recall may overstate the quality of evidence retrieval.

What to investigate when recall is low

Check whether the expected document is indexed and whether its latest version is active. Inspect query rewriting, embedding compatibility, chunk boundaries, search filters, and candidate depth.

If relevant documents are consistently missing, modifying the generation prompt is unlikely to address the underlying problem.

7. Tracing Reranking Failures

Reranking failures are more subtle because the correct evidence may already exist in the retrieved candidates.

Consider this example:

Initial Retrieval: Rank 1: database_manual Rank 2: security_policy_v3 Rank 3: access_guidelines Rank 4: credential_faq Rank 5: deployment_notes 

After reranking:

Rank 1: database_manual Rank 2: deployment_notes Rank 3: credential_faq Rank 4: access_guidelines Rank 5: security_policy_v3 

If the application sends only the top three passages to the LLM, the correct policy is excluded. Retrieval succeeded in finding the relevant document. Reranking failed to preserve it within the selected context.

Compare candidate rankings before and after reranking

Every reranking trace should retain candidate identities so engineers can compare positions across stages.

A simple diagnostic function can identify which documents were dropped.

def dropped_candidates( retrieved_ids, reranked_ids, top_k ): selected = set( reranked_ids[:top_k] ) return [ document_id for document_id in retrieved_ids if document_id not in selected ] 

Example:

retrieved = [ "database_manual", "security_policy_v3", "access_guidelines", "credential_faq" ] reranked = [ "database_manual", "credential_faq", "access_guidelines", "security_policy_v3" ] dropped = dropped_candidates( retrieved, reranked, top_k=3 ) assert "security_policy_v3" in dropped 

This function does not determine whether the reranker is correct. It identifies candidates excluded by the cutoff, allowing an evaluation system to compare those candidates with known relevant evidence.

Common reranking problems

Query mismatch: The reranker receives a rewritten query that no longer represents the user's original information need.

Truncation: The reranker processes only part of a long document chunk, potentially losing the answer-bearing passage.

Domain mismatch: A general-purpose reranker may not rank specialized technical or legal documents as expected.

Aggressive filtering: A small final top-k value can remove useful evidence even when the initial candidate set has good recall.

Reranker scores should not be interpreted as calibrated probabilities unless the model documentation and evaluation support that interpretation.

8. Tracing Generation Failures

Generation debugging begins after the relevant evidence has successfully reached the model.

Suppose the final context contains:

Production database credentials must be rotated through the approved secrets-management service.

However, the LLM responds:

Database administrators should manually update credentials through the management console.

In this case, retrieval and reranking may have worked correctly.
The generation model produced an answer that contradicts the supplied evidence.

Capture the actual generation context

A generation trace should identify the exact document chunks included in the prompt.

Recording only the retrieval results is insufficient because context assembly may apply additional filtering, truncation, deduplication, or ordering.

For example:

{ "generation_context": { "chunk_ids": [ "policy_v3_chunk_4", "policy_v3_chunk_5" ], "prompt_version": "rag_prompt_v7", "context_token_count": 1450 } } 

This metadata allows engineers to reproduce the prompt construction process when the underlying documents and templates are versioned and retained appropriately.

Evaluate answer faithfulness

Faithfulness measures whether the answer's factual claims are supported by the supplied context.

A practical evaluation process is:

• Extract factual claims from the generated answer.
• Compare each claim against the supplied evidence.
• Identify unsupported or contradictory claims.
• Aggregate the results using a documented scoring method.

LLM-based evaluators can assist with this process, but their judgments should not be treated as infallible.

For important applications, use human-reviewed examples and evaluate the evaluator itself.

A response can be faithful to retrieved context while still being factually outdated if the source material is stale.

9. Connecting Retrieval, Reranking, and Generation Into One Diagnostic Workflow

Stage-level tracing becomes most valuable when engineers can compare evidence across the complete request.

Consider three failure cases.

Case A: Relevant evidence never enters retrieval

The correct document is absent from the initial candidate set. The likely investigation area is document ingestion, query processing, embedding configuration, indexing, or search filtering.

Case B: Relevant evidence is lost during reranking

The correct document appears in the retrieval results but is excluded from the final selected context.

The likely investigation area is reranker configuration, candidate truncation, ranking behavior, or the final context cutoff.

Case C: Relevant evidence reaches the LLM, but the answer is wrong

The correct evidence appears in the generation context, yet the response is unsupported or contradictory.

The likely investigation area is prompt construction, context interpretation, model behavior, or output validation.

This produces a useful debugging sequence:

Incorrect RAG Answer | v Was the correct evidence retrieved? | +----+----+ | | No Yes | | v v Debug Did it survive Retrieval Reranking? | +----+----+ | | No Yes | | v v Debug Debug Context Reranker and Generation 

The sequence is intentionally evidence-first. It avoids blaming the generation model before verifying what information the model actually received.

10. Measuring RAG Quality Across All Three Stages

A single overall quality score can hide important regressions. A production evaluation system should measure each stage independently.

Retrieval metrics

Use Recall@k, hit rate, and retrieval latency.

Where relevance judgments support it, additional ranking metrics can help evaluate the ordering of initial candidates.

Reranking metrics

Use metrics such as Mean Reciprocal Rank (MRR) and normalized Discounted Cumulative Gain (nDCG).

MRR is useful when identifying the position of the first relevant result.
nDCG is useful when documents have graded relevance judgments.

Also track whether known relevant evidence survives the final top-k cutoff.

Generation metrics

Evaluate answer faithfulness, answer correctness against reference answers, citation support, and response latency.

Track token usage and model errors separately.

End-to-end metrics

Measure successful answer rate on a representative evaluation dataset, total request latency, cost per request, and authorization correctness.

A RAG application can improve one metric while degrading another.

For example, increasing retrieval depth may improve candidate recall while increasing reranking latency.

Increasing the number of context passages may preserve more evidence but also introduce irrelevant information or exceed context budgets.

These trade-offs should be evaluated using representative workloads rather than assumed.

11. Production Observability: Security, Cost, and Scalability

Tracing introduces operational responsibilities of its own.

Protect sensitive content

Enterprise RAG systems may process confidential documents, employee information, customer records, and internal business data.

Raw prompts and retrieved passages should not automatically be exported to third-party tracing systems.

Prefer document identifiers, version references, counts, model metadata, and carefully controlled evaluation artifacts.

When full-content tracing is necessary, apply explicit access controls, retention limits, and redaction policies.

Control telemetry volume

Recording every retrieved chunk and every model response can create significant storage overhead.

A practical strategy combines:

• Low-overhead metadata for routine production requests.
• Detailed traces for errors and selected diagnostic sessions.
• Representative sampling for performance analysis.
• Dedicated evaluation runs with approved test datasets .

Sampling should preserve enough information to investigate important failures.

For distributed traces, ensure that trace context propagates across retrieval services, reranking workers, model gateways, and asynchronous tasks.

Monitor latency by stage

End-to-end latency alone cannot reveal whether a regression originates in search, reranking, or generation.

Record separate latency measurements for each stage.

If a reranker becomes slower after a model upgrade, stage-level traces make that change visible without requiring changes to the retrieval system.

Handle tracing failures safely

Observability infrastructure should not become an unnecessary dependency for answering user requests.

Where appropriate, telemetry export should be asynchronous and bounded. Applications should define how to handle exporter outages, dropped spans, and backpressure without exposing sensitive information or destabilizing the serving path.

12. Tools for Tracing Production RAG Applications

Several established tools and specifications can support this workflow.

OpenTelemetry

OpenTelemetry provides vendor-neutral tracing infrastructure and context propagation.

It is useful when RAG operations span multiple services and need to integrate with existing application observability.

OpenInference

OpenInference defines AI-oriented tracing conventions, including explicit Retriever, Reranker, LLM, Embedding, and Agent span categories.

It can help represent AI application operations consistently across compatible instrumentation and observability tools.

LangSmith

LangSmith provides tracing and evaluation capabilities for LLM applications, including inspection of retrieval and generation behavior.

It is particularly relevant for teams using LangChain or LangGraph, although integration choices should depend on the existing application architecture.

13. Where AI Engineering Partners Fit Into RAG Observability

For organizations developing production-ready AI applications, RAG observability is part of broader enterprise AI architecture rather than an isolated debugging feature.

CodingCops is an AI-native technology and engineering partner providing AI development and consulting services, including custom LLM applications, RAG solutions, AI agents, enterprise AI integrations, and intelligent automation.

These capabilities are relevant to organizations designing RAG systems that need secure data access, reliable retrieval, and integration with existing enterprise software.

Regardless of whether the work is handled internally or with an engineering partner, tracing requirements should be defined during system design rather than added only after production failures become difficult to diagnose.

14. A Practical Checklist Before Deploying RAG Tracing

Before enabling production tracing, verify the following:

• Every RAG request has a trace identifier.
• Retrieval, reranking, context assembly, and generation have separate spans.
• Document and chunk identifiers remain consistent across stages.
• Retrieval candidate counts, rankings, and index versions are recorded.
• Reranker input and output document identifiers can be compared.
• The exact context selected for generation can be reconstructed securely.
• Retrieval recall is evaluated against known relevant evidence.
• Reranking quality is evaluated independently.
• Generated answers are checked for evidence support.
• Sensitive prompts and document content are excluded from default telemetry.
• Trace context propagates across services and asynchronous workers.
• Telemetry export failures do not destabilize the application.
• Model, prompt, and index configuration versions are recorded.
• Regression tests run before major retrieval or model changes.

Conclusion

RAG failures are difficult to debug when retrieval, reranking, and generation are treated as one operation. An incorrect answer does not automatically indicate a language-model problem.

The relevant document may never have been retrieved. A reranker may have discarded it. Context assembly may have truncated it. Or the generation model may have ignored evidence that was available.

Tracing makes these distinctions observable. By preserving candidate identities, ranking changes, selected context, model configurations, and stage-level evaluation results, engineering teams can identify where a RAG pipeline fails and test targeted improvements.

For production systems, the next practical step is to instrument one complete request path and build a small evaluation dataset containing known retrieval, reranking, and generation failures.

That provides a reproducible foundation for improving RAG reliability without relying on guesswork.