A question comes in: "What is our notice period for contractors?"

Two searches run at once. One looks for meaning, so "contractors leaving" finds "notice period". One looks for exact words, so a product code or an invoice number is not blurred. The two ranked lists are merged, the best passages go to the model, and the answer comes back with a source next to every claim.

This post is the implementation. We built it inside Keel, our agentic AI platform, and it runs on a single Postgres database with the pgvector extension. No separate vector database, no third party embedding API.

The pipeline

Five steps from a document to a cited answer.

  1. Parse: text from PDF, Word, HTML, markdown, plain text. Scans go through OCR first.
  2. Chunk: recursive split into passages of about 800 characters with 120 characters of overlap.
  3. Embed: each chunk becomes a vector, computed where the system runs.
  4. Index: the vector goes into pgvector, the text goes into a full-text index, same database.
  5. Search and merge: a question runs both searches and the two ranked lists are merged.

Everything after that is the agent reading the merged passages and writing from them.

Chunking

Small enough to be specific, large enough to keep a clause together with its heading. The overlap exists because answers rarely respect a boundary: a sentence split across two chunks would be half missing from both.

Both numbers belong to the knowledge base, not the code. A contract set and a support ticket archive have different shapes, so they use different sizes without anyone editing the pipeline.

const parts = await md.chunk({ strategy: "recursive", maxSize: kb?.chunkSize ?? 800, overlap: kb?.chunkOverlap ?? 120, });

We replace a document's chunks in one transaction, so a half processed file never serves answers, and a deleted document stops appearing immediately.

Embeddings that stay local

Embeddings are computed on the machine that runs the system, with a small model through FastEmbed. Documents are never sent to a third party just to be indexed. That matters for data residency, for regulated customers, and for installs with no internet at all. It also means reindexing a large archive does not create a per document bill.

const vectors = await embedder.embed(parts.map(p => p.text)); // vectors[i] is a float32 array, length 384 for BGE-small

The schema

One table for documents, one for chunks, both in Postgres.

create table chunks ( id uuid primary key, document_id uuid not null references documents(id) on delete cascade, content text not null, embedding vector(384) not null, search tsvector generated always as (to_tsvector('english', content)) stored, org_id uuid not null, knowledge_base_id uuid not null ); create index chunks_embedding_idx on chunks using hnsw (embedding vector_cosine_ops); create index chunks_search_idx on chunks using gin (search);

The tsvector is generated, so it cannot drift from the content. The HNSW index keeps vector search fast without a separate service.

Hybrid search

Two searches run on every question.

async function search(query: string, kbId: string, orgId: string): Promise<SearchHit[]> { const qvec = await embedder.embed([query]); const vectorHits = await db.execute(sql` select id, document_id, title, content, 1 - (embedding <=> ${qvec[0]}) as score from chunks where knowledge_base_id = ${kbId} and org_id = ${orgId} order by embedding <=> ${qvec[0]} limit 20 `); const textHits = await db.execute(sql` select id, document_id, title, content, ts_rank_cd(search, plainto_tsquery('english', ${query})) as score from chunks where knowledge_base_id = ${kbId} and org_id = ${orgId} and search @@ plainto_tsquery('english', ${query}) order by score desc limit 20 `); return merge(vectorHits, textHits); }

The vector search finds a paraphrase: "contractors leaving" matches "notice period". The text search finds an exact product code, an invoice number or a surname that embeddings blur.

Merging by rank, not score

The two lists are merged with reciprocal rank fusion. You cannot compare a cosine distance and a text rank directly, they are not the same kind of number. RRF scores a passage by where it appears in each list instead.

function rrf(lists: SearchHit[][], k = 60): SearchHit[] { const scores = new Map<string, number>(); const byId = new Map<string, SearchHit>(); for (const list of lists) { list.forEach((hit, i) => { const rank = i + 1; scores.set(hit.id, (scores.get(hit.id) ?? 0) + 1 / (k + rank)); byId.set(hit.id, hit); }); } return Array.from(scores.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 8) .map(([id]) => byId.get(id)!); }

The practical effect: a passage that is second in both lists beats a passage that is first in one and missing from the other. That is usually the right answer.

The constant k (60) dampens the effect of the top ranks. A passage ranked first is not worth infinitely more than one ranked second.

Scoping by permission

Search is filtered by org and knowledge base at the SQL level, not in application code. A person only gets answers from documents they are allowed to read, and the filter cannot be bypassed by a clever prompt.

where knowledge_base_id = ${kbId} and org_id = ${orgId}

The rules the agent runs under

Retrieval is a tool the agent calls. The rules are instructions plus scoping in the query.

  • When the question is about the organisation, search the knowledge base first.
  • Answer only from the passages that came back, not from what the model remembers.
  • Cite each claim inline with the passage number, like [1] or [2], and list the sources at the end.
  • If the passages do not contain the answer, say you could not find it. Do not fill the gap.
  • Search only the knowledge bases the person is allowed to use.

Every search is a tool call, recorded like any other action: who asked, what was searched, which passages came back, what the agent did with them. When an answer is questioned a month later, the evidence is still there.

Getting documents in, and keeping them current

Documents arrive by upload, from a connected drive or wiki, or by pointing the system at a website. Website import reads the sitemap or a list of pages, so a company's public help pages can become a knowledge base in minutes rather than a migration project.

Every document carries a status: ready with a chunk count, or failed with the reason. Reindexing replaces a document's chunks in one transaction. A deleted document stops appearing immediately.

When RAG is the wrong tool

RAG answers questions from text. It is the wrong choice in three common cases.

  • The answer is a number that changes today, like stock levels, balances or open tickets. That is a database query, not a document search.
  • The job is an action, like issuing a refund, booking a slot or opening a ticket. That is a workflow with permissions, not a chat answer.
  • You need arithmetic over a table. Retrieval gives you the rows, but something still has to add them up.

A useful assistant usually does all three. It searches documents, queries the database and calls tools. The agent decides which one a question needs.

Testing

Write down ten questions you already know the answer to, and the document each answer lives in. Ask them. Count how many answers are right and how many cite the right document. Ask three questions your documents cannot answer. Every one should get a refusal, not a guess. Change one document and ask again. If the answer does not move, indexing is broken.

You will learn more from those twenty minutes than from a month of demos.

We build AI agents on your data and tools, ship them to production and run them, all on Keel. Send us a problem at infrenic.com/free-ai-poc and see it working in 48 hours.