My Support-FAQ Bot Made Up 22 Answers It Didn't Know. Grounding It in n8n Dropped That to 0

A support bot that says "I don't know" is annoying. A support bot that confidently says "No, we don't have a phone number" when nobody ever wrote that down is a liability.

So I tested mine. I gave a small support-FAQ workflow a 6-entry knowledge base and asked it 20 questions: 10 the knowledge base answers, and 10 it doesn't. Then I asked every question three times, to a "raw" version and a "grounded" version.

Setup: n8n 1.123.84 self-hosted, Qwen 1.8B (Q4_K_M) in Ollama on CPU through its OpenAI-compatible /v1 endpoint, temperature 0.2, response_format: json_object. Two workflows: Webhook, Config, Build Prompt, LLM, a Code node, Respond. 20 questions x 3 rounds x 2 versions = 120 runs. Average latency 1.4 s.

The knowledge base is tiny on purpose: refunds (14 days, one email address), pricing (Starter $19, Pro $49, no annual billing), support hours, a 7-day trial with no card, cancellation, and CSV export for Pro. The 10 questions it can't answer include a student discount, a phone number, a Slack integration, GDPR, adding team members, an iPhone app, an uptime SLA, GST invoices, password reset, and a refund after 30 days.

The raw version

The prompt is what most tutorials ship: "You are a friendly support agent. Use the knowledge base to help the customer. Return JSON {answer, escalate}. Set escalate=true only if you cannot help."

On known questions it was perfect. That's exactly why it's dangerous: you test it with the questions you know the answer to, it passes, and you ship it.

I read all 29 answers it gave to questions it had no answer for. 22 of them asserted something that isn't in the knowledge base:

• "No, we do not offer a student discount." (3/3 runs)
• "No, there is no phone number to call for support." (3/3)
• "No, we do not integrate with Slack." (3/3)
• "No, we do not have a mobile app for iPhone." (3/3)
• "We do not have an uptime guarantee or SLA." (2 of 3 runs)
• "Resetting your password is not supported by YourCo. Please contact support@yourco.com with your order number to initiate a refund." (3/3 runs gave this kind of answer; once the model flagged it for escalation, the other 2 it sent as is. It's a password question, and the refund line is just the nearest KB text)
• "Pro users can add team members for $49/mo", and in another run, "adding team members is free, through Settings > Team Members". There is no such menu.

Some of those "no" answers might even be true. That's not the point. Nobody told the bot, and nobody checked. A customer who hears "you can't have that" from a support bot doesn't buy.

The other 7: three answered the GDPR question with the CSV export entry (off-topic, not an invention), three answered "refund after 30 days?" with "no, 14 days only" (which is what the KB implies, so I counted it as fine), and one said "not in the knowledge base" but didn't escalate.

Grounded v1: make the model prove it, then check in code

The fix isn't a sterner prompt. My grounded prompt also says "answer ONLY from the knowledge base", and a 1.8B model still wrote answers that weren't in it. The fix is to make the model hand over evidence, and have a Code node check it.

The prompt asks for {answer, kb_id, quote, escalate}: which entry it used, and the sentence it copied from it. The Code node then decides:

const KB = { refunds: "Refunds: full refund within 14 days...", pricing: "...", /* one entry per id */ }; const FALLBACK = 'I am not sure, a teammate will follow up shortly.'; const norm = s => String(s||'').toLowerCase().replace(/[^a-z0-9]/g,''); if (out.escalate === true || out.kb_id === 'none') reasons.push('model escalated'); else if (!KB[out.kb_id]) reasons.push('kb_id not in knowledge base'); else { const entry = KB[out.kb_id]; if (!out.quote || !norm(entry).includes(norm(out.quote))) reasons.push('quote not found in cited entry'); // every number and email in the answer must appear in the cited entry const facts = String(out.answer||'').match(/\$?\d+[\d.,]*|[\w.+-]+@[\w.-]+/g) || []; const bad = facts.filter(f => !norm(entry).includes(norm(f))); if (bad.length) reasons.push('facts not in cited entry: ' + bad.join(' ')); } // anything that fails gets the fallback answer and escalate: true 

Zero made-up answers. But look at the last row: it threw away 18 of 30 correct answers.

The model is bad at filling in bookkeeping fields, not at answering. It wrote kb_id: "refund" (the id is refunds), "free_trial" (the id is trial), and often kb_id: "none" while quoting the right sentence word for word. A guard that trusts the model's own labels punishes the model for typos.

It also did catch the model faking evidence. On the questions it didn't know, the "quote" was often just the answer it had made up, copied into the quote field: "Reset your password by going to Settings > Passwords and clicking on 'Forgot Password?'" The quote check failed it every time, and the customer got the fallback.

Grounded v2: ignore the labels, trust only the quote

Same prompt, same model outputs. I changed the Code node to ignore kb_id and the model's escalate flag entirely and only ask: is this quote a real stretch of the knowledge base?

const q = norm(out.quote); const hit = q.length >= 20 ? Object.values(KB).find(e => norm(e).includes(q)) : null; if (!hit) reasons.push('quote not found in knowledge base'); else { const facts = String(out.answer||'').match(/\$?\d+[\d.,]*|[\w.+-]+@[\w.-]+/g) || []; const bad = facts.filter(f => !norm(hit).includes(norm(f))); if (bad.length) reasons.push('facts not in quoted entry: ' + bad.join(' ')); } 

I replayed the 60 saved grounded outputs through it (no new model calls):

The 3 that "got through" were the 30-day refund question, answered "full refunds are only available within 14 days of purchase", a real quote from a real entry. That's the correct answer, so I'd call it a pass. The 4 wrongly escalated were the model paraphrasing the quote instead of copying it, and falling back to a human is a cheap failure.

What this check can't do

It checks that the quote exists, not that it answers the question. A model that quotes the right entry for the wrong question will pass. The GDPR question was answered with the CSV export sentence in the raw version, and a quote check wouldn't stop that if the model quoted the export entry accurately. In my run it didn't get that far, because the model wrote its own text into the quote field, but I wouldn't rely on luck. For that you'd add a keyword overlap check between the question and the entry, or a human spot-check on the first 50 answers.

Also, with only 6 KB entries and 1.8B parameters, the results will be different on your KB and your model. The 22 isn't a rate you should expect. What carries over is the method: ask the model for evidence, and let code decide whether the evidence is real.

What I'd do with any n8n support bot

1. Test with questions your KB can't answer. Half of my test set was "not in the KB". The raw bot scored 100% on the other half.
2. Count escalations, not just answers. Raw: 1 escalation out of 30 unknowns. A bot that never says "I don't know" isn't confident, it's broken.
3. Make the model quote its source and check the quote in code. Prompts alone didn't stop it.
4. Don't trust the model's own labels. Ids, flags and "confidence" fields were wrong far more often than the answers. Check the evidence, not the labels.
5. Make the fallback boring. "A teammate will follow up shortly" costs you a delayed reply. A wrong "no" costs you the customer.
6. Measure the cost of the guard. v1 blocked 18 correct answers and looked perfect on made-up ones.

The free workflows

The support-FAQ workflows from this test aren't in the free repo yet, and the Code node above is the important part: paste it after any LLM node that answers from a knowledge base. The free workflows in arkintof/n8n-ai-agent-workflows right now are the lead qualifier and the review reply bot, and the idea carries over to both: let the model propose, let code decide.

What's the most confident wrong answer your support bot has ever given a customer? Did you find out from a log, or from the customer?

Written with AI assistance; workflow built and tested by me.