Connecting an LLM to your wiki, test manager, and database for QA

Most QA questions I get — "why did this suite fail last sprint", "does the wiki mention a rate limit for this endpoint", "which cases cover the payment retry path" — have an answer somewhere, just never in the tool I'm currently looking at. The wiki, the test management system and the results database each hold a third of the picture, and keyword search across them is close to useless.

So I built a small connector that pulls all three into PostgreSQL with pgvector and lets an LLM answer from retrieved chunks. That's plain RAG: the model retrieves relevant documents first, then answers from them. Hands-on time for me was about 90 minutes; the ingestion run itself depends on how many pages and test cases you have.

What you need before touching the code

Here's the shopping list I worked from:

• OS: Linux, macOS, or Windows with WSL2 — every command below runs in bash.
• Docker + Docker Compose so PostgreSQL with the vector extension runs in a container instead of being compiled on the host.
• Python 3 with requests, psycopg2-binary, fastapi, uvicorn[standard].
• Read-only credentials: a wiki API token, a test management user + API token, and a dedicated database role that can only SELECT.
• LLM access: base URL, API key, one chat model name, one embedding model name. Take the exact model names and the embedding dimension from your provider's docs — I'm leaving them as placeholders on purpose, because a wrong dimension is the most common reason the first insert fails.
• Network: outbound HTTPS to all three sources plus the LLM endpoint, and one free local port (8080 in this write-up).

mkdir -p /opt/qa-connector && cd /opt/qa-connector python3 -m venv .venv && source .venv/bin/activate pip install requests psycopg2-binary fastapi "uvicorn[standard]" 

Sanity check: pip show fastapi should print a Location inside .venv. If that path points at the system Python, the virtualenv isn't active — the system interpreter can be upgraded under you and break the job silently later.

Step 1: map each source and scope its token

Before writing code I wrote one row per source: what I pull, which endpoint, which auth scheme.

Then the role, the database and the extension:

CREATE ROLE qa_ai LOGIN PASSWORD '<db-password>'; CREATE DATABASE qa_knowledge OWNER qa_ai; \c qa_knowledge CREATE EXTENSION IF NOT EXISTS vector; GRANT USAGE ON SCHEMA public TO qa_ai; 

And the .env file — never committed, obviously:

WIKI_BASE_URL=https://<your-domain>/wiki WIKI_TOKEN=<wiki-api-token> WIKI_SPACE_KEY=QA TESTMGMT_BASE_URL=https://<your-domain>/index.php?/api/v2 TESTMGMT_USER=qa-bot@example.com TESTMGMT_TOKEN=<test-management-api-token> TESTMGMT_PROJECT_ID=1 PGHOST=127.0.0.1 PGPORT=5432 PGDATABASE=qa_knowledge PGUSER=qa_ai PGPASSWORD=<db-password> LLM_BASE_URL=https://<your-llm-endpoint>/v1 LLM_API_KEY=<llm-api-key> LLM_MODEL=<chat-model-name> EMBEDDING_MODEL=<embedding-model-name> 

One credential per source with the narrowest scope means a leaked wiki token can't rewrite a test case, and a SELECT-only role can't damage production data. Verify the pieces before moving on: the psql call should list plpgsql and vector, and the curl should return 200.

psql -h 127.0.0.1 -U qa_ai -d qa_knowledge -c "select extname from pg_extension;" curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $WIKI_TOKEN" \ "$WIKI_BASE_URL/rest/api/content?type=page&limit=1" 

401 means the token is wrong, 404 means the base path is wrong — in my case I'd forgotten the /wiki suffix.

Step 2: create the table that holds chunks and vectors

One table serves all three sources. The unique constraint is what makes re-runs safe.

CREATE TABLE documents ( id bigserial PRIMARY KEY, source text NOT NULL CHECK (source IN ('wiki','testmgmt','db')), external_id text NOT NULL, title text NOT NULL, url text, chunk_no int NOT NULL DEFAULT 0, body text NOT NULL, updated_at timestamptz NOT NULL DEFAULT now(), embedding vector(<dim>), -- <dim> from your embedding model's docs UNIQUE (source, external_id, chunk_no) ); CREATE INDEX documents_embedding_idx ON documents USING hnsw (embedding vector_cosine_ops); 

Swap <dim> for the real dimension. If your pgvector build is too old for hnsw, drop the index line and check the pgvector docs for alternatives.

Step 3: chunk, embed, upsert

The ingestion script has the same shape for every source; here's the wiki half. Pages get cut into overlapping windows, sent to the embedding endpoint in batches, and written with ON CONFLICT so a second run updates pages instead of duplicating them.

import os, requests, psycopg2 from psycopg2.extras import execute_values from dotenv import load_dotenv; load_dotenv() WIKI_BASE = os.environ["WIKI_BASE_URL"].rstrip("/") HEADERS = {"Authorization": f"Bearer {os.environ['WIKI_TOKEN']}", "Accept": "application/json"} def iter_pages(): url = f"{WIKI_BASE}/rest/api/content" params = {"type": "page", "spaceKey": os.environ["WIKI_SPACE_KEY"], "limit": 50, "expand": "body.storage,version"} while url: r = requests.get(url, headers=HEADERS, params=params, timeout=30) r.raise_for_status() data = r.json() yield from data["results"] nxt = data.get("_links", {}).get("next") url = f"{WIKI_BASE}{nxt}" if nxt else None params = None def chunk_text(text, size=1200, overlap=200): for i in range(0, len(text), size - overlap): yield text[i:i + size] def embed_batch(texts): r = requests.post(f"{os.environ['LLM_BASE_URL']}/embeddings", headers={"Authorization": f"Bearer {os.environ['LLM_API_KEY']}"}, json={"model": os.environ["EMBEDDING_MODEL"], "input": texts}, timeout=60) r.raise_for_status() return [d["embedding"] for d in r.json()["data"]] rows = [] for page in iter_pages(): body = page["body"]["storage"]["value"] for i, piece in enumerate(chunk_text(body)): rows.append(("wiki", page["id"], page["title"], f"{WIKI_BASE}/spaces/{page['space']['key']}/pages/{page['id']}", i, piece)) with psycopg2.connect(host=os.environ["PGHOST"], port=os.environ["PGPORT"], dbname=os.environ["PGDATABASE"], user=os.environ["PGUSER"], password=os.environ["PGPASSWORD"]) as conn, conn.cursor() as cur: for start in range(0, len(rows), 50): batch = rows[start:start + 50] vectors = embed_batch([r[5] for r in batch]) execute_values(cur, """ INSERT INTO documents (source, external_id, title, url, chunk_no, body, embedding) VALUES %s ON CONFLICT (source, external_id, chunk_no) DO UPDATE SET body = EXCLUDED.body, embedding = EXCLUDED.embedding, updated_at = now() """, [b + (v,) for b, v in zip(batch, vectors)], template="(%s,%s,%s,%s,%s,%s,%s::vector)") 

The ::vector cast is what turns a Python list into something Postgres accepts, and batching at 50 keeps the embedding request under typical payload limits. Repeat the same loop for the test management project and for your SQL views — only the fetch function changes.

Step 4: verify before you schedule anything

Run the job once, then measure. Row counts per source tell you whether the pagination actually worked, and they also give you a realistic estimate of ingestion time and disk usage before you put the job on a cron.

python ingest.py psql -h 127.0.0.1 -U qa_ai -d qa_knowledge \ -c "select source, count(*) from documents group by source order by source;" 

If the wiki row count looks suspiciously round (50, 100), pagination stopped early — check the _links.next handling first. Once counts look right, the retrieval endpoint on uvicorn (port 8080) is a thin wrapper: embed the question, order by cosine distance, feed the top chunks to the chat model.

Takeaway

The hard part of this project wasn't the LLM — it was writing down three endpoints and scoping three credentials. Once the vectors live next to the rest of your data in one table with one unique key, re-running ingestion becomes boring, and boring is exactly what you want from a system your QA team depends on.

This article was originally published on AI Release — https://ai-release.org — and I post shorter notes and updates on Telegram at https://t.me/ai_release1.