Short answer: build the healthcare appointment assistant so every answer carries a machine-checkable path from the user question to retrieved source revisions, selected passages, and sentence-level citations; reject an answer when that path is incomplete.

For appointment questions, a fluent response is a weak success criterion. The useful criterion is narrower: can an operator reconstruct why the assistant said a clinic accepts walk-ins, where that statement came from, and which version of the scheduling policy was searched? The least complex architecture that meets that bar is a retrieval pipeline with an append-only trace beside the generated response.

The data flow is plain: normalize the question, retrieve versioned passages, assign stable citation labels, generate only from those passages, validate the labels, then persist the question, retrieval results, answer, and source revisions under one trace ID. This keeps the notebook experiment recognizable when it becomes a service.

How should a healthcare appointment assistant trace retrieval citations?

Trace two related sets instead of collapsing them into one. The retrieval set records every passage considered, including its score and revision. The citation set records only passages the final answer actually invokes. That distinction matters because a high-ranking result may be context for generation without supporting any sentence in the response. Treating all retrieved passages as citations creates a comforting audit log that cannot answer the hard question: which source supports this claim?

Keep both sets.

Use stable source IDs and immutable revision IDs. A URL alone is not enough for the design because its contents can change while the address stays fixed. Chunk offsets are useful debugging metadata, but they should sit below a durable document-and-revision identity; re-chunking must not make an old trace appear to reference new text. For each answer sentence that states an appointment rule, require at least one citation label, and resolve that label to the exact passage supplied to the generator.

A compact trace can contain trace_id, query, retrieved, answer, citations, and created_at. Keep the raw similarity score for evaluation, not as proof that a claim is true. Retrieval answers relevance; citation validation answers provenance. They are different jobs.

This is strict by design.

The trace also needs an explicit abstention path. If no retrieved passage supports the requested scheduling detail, the assistant should say it cannot verify the detail from its indexed material and direct the user to the normal appointment channel. Don't let a model fill a missing clinic rule from plausible language. Consider a rescheduling question that retrieves a clinic-hours page first and a scheduling-policy page second: the hours page may share more query words, yet it says nothing about who can change a booking. The assistant can cite the second passage if it supports the instruction, but citing the first merely because it ranked first would hide the grounding failure. In an eval harness, that unsupported-answer case deserves its own label — separate from generic accuracy — so a ranking change cannot quietly turn missing evidence into a polished instruction.

Build the trace before adding model complexity

Here is a runnable Python sketch. It uses a tiny lexical retriever so the provenance mechanics stay visible; a production retriever can replace retrieve without changing the trace contract. The example records all candidates, exposes labeled context to the answer function, and refuses unknown or missing citation labels.

from __future__ import annotations from dataclasses import asdict, dataclass from datetime import datetime, timezone import json import re import uuid from typing import Callable @dataclass(frozen=True) class Passage: source_id: str revision: str text: str @dataclass(frozen=True) class Hit: label: str source_id: str revision: str text: str score: int def terms(value: str) -> set[str]: return set(re.findall(r"[a-z0-9]+", value.lower())) def retrieve(query: str, corpus: list[Passage], limit: int = 3) -> list[Hit]: query_terms = terms(query) ranked = sorted( corpus, key=lambda passage: len(query_terms & terms(passage.text)), reverse=True, ) hits = [] for index, passage in enumerate(ranked[:limit], start=1): hits.append( Hit( label=f"S{index}", source_id=passage.source_id, revision=passage.revision, text=passage.text, score=len(query_terms & terms(passage.text)), ) ) return hits def validate_citations(answer: str, hits: list[Hit]) -> list[str]: used = re.findall(r"\[(S\d+)\]", answer) known = {hit.label for hit in hits} unknown = sorted(set(used) - known) if unknown: raise ValueError(f"CITATION_UNKNOWN: {unknown}") if not used: raise ValueError("CITATION_GAP: answer has no source label") return sorted(set(used)) def answer_with_trace( query: str, corpus: list[Passage], generate: Callable[[str, str], str], ) -> dict: hits = retrieve(query, corpus) context = "\n".join( f"[{hit.label}] source={hit.source_id} revision={hit.revision}\n{hit.text}" for hit in hits ) answer = generate(query, context) used_labels = validate_citations(answer, hits) by_label = {hit.label: hit for hit in hits} return { "trace_id": str(uuid.uuid4()), "created_at": datetime.now(timezone.utc).isoformat(), "query": query, "retrieved": [asdict(hit) for hit in hits], "answer": answer, "citations": [asdict(by_label[label]) for label in used_labels], } corpus = [ Passage( source_id="clinic-north-scheduling", revision="2026-08-20", text="North Clinic rescheduling requests are handled by the scheduling desk.", ), Passage( source_id="clinic-north-hours", revision="2026-08-18", text="North Clinic appointment hours are listed in the booking system.", ), ] def demo_generator(query: str, context: str) -> str: # A real generator receives the same labeled context and citation contract. return "Contact the scheduling desk to request rescheduling. [S1]" trace = answer_with_trace( "How do I reschedule at North Clinic?", corpus, demo_generator ) print(json.dumps(trace, indent=2))

The two all-caps error labels are local application decisions, not transport status codes. They make failed evaluations easy to group: CITATION_GAP means the answer supplied no evidence label, while CITATION_UNKNOWN means it named evidence that was never retrieved. A service can map both to an abstaining user response while preserving the internal reason in the trace.

There is a deliberate limitation in this sample: token overlap is useful for demonstrating the contract, but it is not suitable for real semantic retrieval. Replace it with the retriever selected by your own corpus evaluation. Keep the Hit shape, stable revisions, citation check, and stored trace unchanged, so a retriever experiment does not silently rewrite the audit model.

Evaluate grounding as a pipeline, not a vibe

Start with a small appointment-question set drawn from the kinds of documents the assistant is allowed to use. Each case should specify an expected outcome: supported answer, abstention because support is absent, or clarification because the request lacks a clinic, location, or appointment type. Then score retrieval and answer behavior separately. This is the notebook-to-prod hinge: a single end-to-end score can improve while the system gets worse at finding the right revision, because the generator may compensate with fluent guesses.

For retrieval, inspect whether the supporting revision appears in the candidate set and where it ranks. For generation, inspect whether each operational claim is entailed by at least one cited passage, whether every citation resolves, and whether the answer adds details absent from the context. For the whole system, replay stored traces after changes to chunking, indexing, prompting, or models. The exact thresholds belong to the deployment's risk decision; I'm not sure a universal pass rate would mean much without the corpus, review policy, and cost of a wrong appointment instruction.

A useful failure matrix is small enough to read during review:

Failure Trace signal Preferred behavior
Correct source was not retrieved Expected revision absent from retrieved Abstain, then tune retrieval
Old and current revisions both rank Conflicting revision IDs in candidates Select the approved current revision or abstain
Answer names an unknown label CITATION_UNKNOWN Reject the answer
Answer gives a rule without evidence CITATION_GAP or evaluator finding Reject or regenerate from supplied context
Question is underspecified Required scheduling entity is absent Ask one focused clarification

Prompt cost belongs in this evaluation loop too. Store how many passages were offered to the generator and compare citation coverage as that context changes. More context can expose more possible evidence, but it also increases the material the generator must distinguish. The right context budget is the smallest one that meets retrieval and grounding targets on the held-out cases, not the largest window available. No drama. Measure it.

The catch is that sentence-level labels do not prove the source itself is correct, current, or approved. This architecture is not suitable when the corpus lacks revision control and ownership metadata; fix that information lifecycle first. It is also a poor fit for questions that cross into diagnosis or clinical advice if the assistant is scoped only to appointment operations. Route those outside the appointment-answer path rather than pretending stronger retrieval changes the scope.

Operate the evidence path after deployment

Deployment should preserve the exact inputs needed for replay without turning the trace into an accidental store of unnecessary user data. Decide which query fields are required for debugging, apply the organization's retention and access rules, and keep source text access aligned with the permissions used at retrieval time. Provenance is not permission: a perfectly cited passage can still be inappropriate for a particular user to see.

Watch the failure categories, not just request volume. A rising abstention rate can point to corpus drift, new question patterns, or a retrieval change; an increase in unresolved citations points directly to a contract violation between context assembly and generation. Review samples of successful answers as well, because a validator that checks label syntax cannot decide by itself whether a passage actually entails a sentence. Human review and an automated entailment check can complement the deterministic label validation, but neither should erase the original trace.

Before release, read one stored record from end to end. The query should reproduce the retrieved source IDs and revisions; every label in the answer should resolve to supplied text; unsupported cases should abstain; access checks should apply before context assembly; and a prompt or retriever change should be testable against the same evaluation set. Then run the replay suite as part of deployment, monitor each named failure mode, and assign someone to source freshness. That's the operational checklist. It is intentionally prose-sized because the owner and decision matter more than another box-ticking artifact.

A retrieval-augmented design gives a generator access to retrieved non-parametric material alongside its learned parameters, as described in the original RAG paper. Citation tracing is an application-level contract built on top of that pattern. Keep the contract independent of the retriever and generator, and the assistant can change components while old appointment answers remain explainable.

References

Further reading