In healthcare data, a confident wrong answer is worse than a slow right one. A retrieval-augmented generation (RAG) demo can look finished in an afternoon. The same system can fail the moment it meets messy identifiers, stale documents, or a question it should have refused.

I am Ocean Coldiron, a Senior Sustaining and Forward Deployed Engineer at Abacus Insights, starting November 16, 2026. I have more than 15 years in healthcare technology and production systems. In that work I have shipped 125 healthcare integrations, imported data for more than 5 million patients, and migrated 100 TB of historical data. I also build AI agents, Model Context Protocol (MCP) tools, and RAG workflows, and I write about the parts that have to hold up once real users depend on them. More about my work is at oceancoldiron.com.

This article is about evaluation, not model worship. The habits below are the ones I want in place before a retrieval system is trusted near healthcare data.

Why healthcare data changes the bar

Healthcare interoperability, including HL7 and FHIR, makes retrieval failures expensive. Records are split across systems. Identifiers collide. Documents go stale and still remain searchable. A model that sounds sure can stitch two true fragments into a wrong answer, or it can answer a question that should have been declined.

I do not treat a fluent answer as evidence that the system is ready. Fluency is cheap. Faithfulness is not. I treat evaluation as the work that happens before anyone trusts the output, not as a polish step at the end.

That posture comes from integration work as much as from model work. When data is late, duplicated, or mapped to the wrong field, the failure is often quiet until someone acts on it. A RAG system can fail the same way: the answer reads cleanly, the citation looks plausible, and the retrieved text is not actually the right record for the question that was asked.

A simple evaluation loop

The loop I use is small on purpose. It has to be something I will actually rerun when a prompt, a model, or a corpus changes.

1. Build a versioned question set. I keep three buckets:

  • Questions the system should answer, with a short note about what a good answer must include and which source it should come from.
  • Questions the system should refuse, including requests outside the corpus, requests for advice the system is not allowed to give, and questions that look like the right domain but do not have support in the knowledge base.
  • Questions that need a citation, where a bare assertion, even if it happens to be right, is not enough.

The refuse set matters as much as the answer set. If you only test happy-path questions, you will not see the system invent a confident reply to something it should not touch.

2. Score retrieval and answers separately. First, did we fetch the right source? Second, did the answer stay faithful to that source? A correct-sounding answer grounded in the wrong document is still a miss. So is a right document followed by a paraphrase that changes the meaning.

3. Keep a regression set. When I change a prompt, a chunking rule, a retriever setting, or a model, I rerun earlier cases. A new improvement that silently breaks an old refusal or citation is not an improvement. The set should be versioned with the code or config that produced it, so I can tell what changed and when.

4. Log traces, not sensitive data. For each run I want to see the retrieved chunks, the tool calls, and the final answer. I do not want patient data, credentials, or anything that does not belong in a log. If a tool was called, I want to know which tool, with what arguments, and whether the call should have been allowed.

This is not a vendor-specific recipe. It is a habit: measure retrieval, measure faithfulness, measure refusals, and keep the history.

What I will not automate away

Some behavior should stay conservative even when the model is capable.

Human review still belongs on high-impact answers. If a response could change a real-world action, a person should be in the loop until the evaluation set, the traces, and the access controls say otherwise. "The model usually gets this right" is not the same as "this answer is safe to use."

Clear "I don't know" behavior is a feature. I would rather the system refuse than fill a gap with a fluent guess. That is especially true when the retrieved context is thin, conflicting, or off-topic. A refusal can be written in plain language: we do not have a source for that, or this question is outside what the system is allowed to answer.

Tool access needs hard boundaries. An agent with MCP tools can look helpful and still be one bad call away from touching the wrong system. I treat tool permissions the way I would treat production credentials: least privilege, explicit allowlists, and traces I can read back later. If I cannot explain why a tool was callable in that moment, it should not have been callable.

A public example, not an employer case study

I keep a small, public example on my site: a RAG-powered assistant on oceancoldiron.com. It runs on a Go backend, uses Claude, and answers from a curated markdown knowledge base with guardrails and prompt caching. I have also built MCP and FastMCP servers that expose tools to agents with tighter boundaries.

That assistant is a portfolio system. It is not a hospital workflow, a payer platform, or a clinical-decision tool. I point to it because it is public, I can talk about how it is built, and it is a place to practice the same evaluation habits: a known corpus, questions it should answer, questions it should refuse, and logs that do not hold sensitive data.

The lesson I want to carry from public work into production work is not "this architecture is done." It is "do not trust a retrieval system because the demo sounded good."

Takeaways

  • Evaluate before you trust. A fluent answer is not a passing test.
  • Score retrieval and answers separately, and keep a regression set so prompt changes cannot silently break earlier cases.
  • Refuse more often than you think, and make "I don't know" an acceptable, visible outcome.
  • Log traces for chunks and tool calls, without storing sensitive data.
  • If the data is sensitive, design the tools as if a wrong call will be read back to you later.

If you are building RAG or agents around healthcare data, start with the question set and the refusals. The model can wait.

Ocean Coldiron is a Senior Sustaining and Forward Deployed Engineer at Abacus Insights, starting November 16, 2026. He writes about applied AI, MCP, and healthcare data, and he has completed all 48 contiguous U.S. state high points. Site: https://oceancoldiron.com/. Outdoor stories: https://oceancoldiron.com/oceansoutdoors/.