Your RAG chatbot just told an employee they get thirty vacation days. The handbook it retrieved says twenty. Nothing crashed, no test failed, and the answer reads perfectly well. That's how retrieval-augmented generation usually breaks: quietly.

I maintain @devilsdev/rag-pipeline-utils, an open-source RAG library for Node.js built around one idea: you should be able to check every answer against the sources it came from, cheaply, in your app and in CI. This post is a quick tour.

npm install @devilsdev/rag-pipeline-utils

Node.js 22+, with CommonJS, ESM and TypeScript types included.

Catch the made-up sentence

You don't have to build your pipeline with this library to use it. Bring an answer from anywhere (LangChain, Mastra, the Vercel AI SDK, LlamaIndex or your own code) and pass it in with the sources it was given:

import { checkAnswer } from "@devilsdev/rag-pipeline-utils"; const check = await checkAnswer({ answer: "Full-time employees receive thirty days of paid vacation per year. " + "Remote work is allowed two days per week.", sources: [ { pageContent: "Full-time employees receive twenty days of paid vacation per year.", }, { pageContent: "Remote work is allowed two days per week with manager approval.", }, ], }); console.log(check.grounded); // false console.log(check.groundedness); // 0.5 console.log(check.unsupported); // ["Full-time employees receive thirty days of paid vacation per year."]

It checks each sentence against the sources: the words, but also the numbers and the negations, which is where answers tend to go wrong. It runs locally with no API key and makes no model calls, so the same input gives the same verdict every time, offline, in tests.

If you're worried about correct paraphrases being flagged, pass a judge and your LLM is asked about the sentences that failed on wording alone, and nothing else.

A pipeline in a dozen lines

If you do want the library to run the pipeline, createRagPipeline() wires an embedder, a retriever, an optional reranker and an LLM. This one runs offline, with the built-in TF-IDF embedder and an in-memory store:

import { createRagPipeline, LocalEmbedder, MemoryRetriever, } from "@devilsdev/rag-pipeline-utils"; const docs = [ { id: "vacation", content: "Full-time employees get twenty vacation days." }, { id: "remote", content: "Remote work is allowed two days per week." }, ]; const embedder = new LocalEmbedder(); embedder.train(docs.map((doc) => doc.content)); const retriever = new MemoryRetriever(); await retriever.store( docs.map((doc) => ({ ...doc, vector: embedder.embed(doc.content) })), ); const pipeline = createRagPipeline({ embedder, retriever }); const result = await pipeline.run({ query: "How many vacation days?" }); console.log(result.results[0].id); // "vacation"

Swap in OpenAI, Anthropic, Cohere or Ollama for the model, and pgvector, Qdrant, Pinecone or Weaviate for the store, when you're ready. Anything with an embed() or generate() method works too, including Vercel AI SDK models.

Prefer to start from a working app? npm create rag-pipeline@latest scaffolds one for Next.js with the AI SDK, Mastra, or Cloudflare Workers.

Test answers like any other value

RAG regressions don't throw. Someone changes the chunk size, the prompt, the model or the embedder, everything still runs, and the answers drift. So the library ships offline test helpers and matchers for Jest and Vitest:

import { expect, test } from "vitest"; // or Jest's global expect import { createTestPipeline, ragMatchers, } from "@devilsdev/rag-pipeline-utils/testing"; expect.extend(ragMatchers); test("answers vacation questions from the handbook", async () => { const { pipeline } = await createTestPipeline(); const result = await pipeline.run({ query: "How many vacation days do I get?", }); expect(result).toBeGroundedIn(); // every sentence supported by result.results expect(result).toCite("vacation"); expect(result).toContainNoPII(); });

For whole test sets there's a CLI: rag-pipeline eval scores a dataset and fails when answers get worse than a saved baseline, and the rag-eval-action GitHub Action runs it on every pull request and posts the comparison as a comment.

What else is in the box

  • Framework integrations. groundingMiddleware adds retrieval and grounding to any Vercel AI SDK model without changing the rest of your code; there are scorers and guardrails for Mastra.
  • Guardrails. Prompt-injection detection and PII redaction, with published precision and recall, plus access-control filtering before anything reaches the model.
  • Production details. Hybrid search, reranking, token budgets, timeouts and cancellation, retries, caching, and OpenTelemetry (or Langfuse) traces with the evaluation scores attached.
  • The edge. A /edge build for Cloudflare Workers, Vercel Edge and Deno.
  • Claude, via MCP. @devilsdev/rag-mcp-server lets Claude search a folder of your documents and check its own answers against them.
  • Take only what you need. The main package has everything; smaller packages like @devilsdev/rag-eval (about 6 MB installed) are there if install size matters.

Try it

The fastest way in is the answer check on StackBlitz: paste an answer of your own, with no install or API key.

I'd love to hear how you test your RAG apps today, and what you'd want a library like this to catch. Tell me in the comments.