A CAIO's perspective on enterprise knowledge architecture. Full whitepaper (PDF, 29 pages, 11 figures): DOI 10.5281/zenodo.24399422.
Executive summary
Across energy, manufacturing, healthcare and financial services, I keep seeing the same pattern. The model is rarely the problem. Enterprise assistants disappoint because the right business knowledge is not in front of the model when it answers, or because nobody can prove the model was allowed to see what it used. RAG, CAG and KAG are three ways of solving the first problem; governance decides the second. For a CAIO or CTO, choosing between them is less a technology bet than a decision about where knowledge preparation happens, who owns it, and what each correct answer costs.
RAG (Retrieval-Augmented Generation) searches a document collection when a question arrives and passes the model the few passages that matter. It is the usual starting point because it copes with large collections, frequent change and per-user permissions.
CAG (Cache-Augmented Generation) loads a small, stable body of knowledge into the model once, keeps the model's processed state (the KV cache) and reuses it for later questions. It can work well for a shared handbook that is queried all day. It does not suit large, fast-changing or permissioned content.
KAG (Knowledge-Augmented Generation) first turns documents into a graph of entities and relationships linked back to the source text, then answers by planning and running a sequence of lookups and calculations. It earns its higher cost on questions that chain several facts, depend on dates, or apply rules.
Mature programs do not pick one. They build a single governed knowledge platform and route each kind of question to the cheapest path that answers it reliably. That is the direction I recommend, and the rest of this article explains how to get there without overinvesting early.
What leaders should take from the research
Most headline numbers do not survive a reading of the papers. CAG's famous "40x faster" is measured against re-reading the whole corpus for every question, not against RAG. Its accuracy edge over RAG is marginal and disappears as the content grows.
Graphs are an investment for specific questions, not a general upgrade. Graph methods win on questions that chain facts, dates and rules, and lose to plain retrieval on simple lookups.
Bigger context windows do not retire retrieval. Models lose accuracy well before their advertised limit, especially when the answer is not phrased like the question.
No approach is reliable on its own. Even the best systems in public benchmarks miss or get wrong a meaningful share of questions. Evaluation, and the ability to say "I don't know", belong in the design from day one.
Five recommendations for the next twelve months
Fix retrieval before buying anything new. Contextual chunking, hybrid search and reranking cut retrieval failures by more than half in published tests, and most production systems I review have not done all three. This is the cheapest accuracy gain available.
Turn on caching where the terms fit, and make CAG earn its place. Prefix caching is close to free money for stable instructions. For small, stable, shared content, run CAG side by side with a well-tuned RAG pipeline before committing.
Fund a graph only where real questions demand it. Look at what users actually ask. If enough of it is relational, time-bound or rule-bound, start with low-cost graph methods and reserve KAG for rule- and calculation-heavy domains such as compliance, claims or engineering change.
Route rather than choose. A router lets each path be justified, measured and retired on its own business case, which is how you avoid a platform built around last year's favourite technique.
Govern all three paths the same way. Authorize before any knowledge reaches the model, isolate caches per tenant, track where every fact came from, and report cost per correct, authorized answer to the business, not cost per token.
The picture below shows where this article lands: one governed platform in which RAG, CAG and KAG are options chosen by an orchestration layer.
Figure 1. Enterprise knowledge architecture: applications on top, an orchestration and knowledge router choosing RAG, CAG or KAG, and a governed knowledge fabric over enterprise data sources.
Part I — Concepts
1. Why LLMs need external knowledge
When I brief a board, I describe a language model as a brilliant new hire who studied everything public up to a certain date and then stopped reading: fluent and fast, but unaware of your company, your latest policies, and anything that happened since.
What a model "knows" — and what it cannot
Everything a model learned in training is stored in its weights. Researchers call this parametric memory. It has three limits that matter to every enterprise:
| Limit | What it means in practice | Example |
|---|---|---|
| Frozen in time | The model knows nothing after its training cutoff | It cannot tell you last quarter's revised pricing |
| Never saw your data | Contracts, maintenance logs, SOPs, customer records were never in training | It cannot answer "what is our warranty term for Product X?" |
| Cannot cite | Even a correct answer arrives without a source | An auditor cannot verify where the answer came from |
When a model lacks the relevant facts it will often still produce a fluent, confident answer. Unsupported answers of this kind are called hallucination.
The open-book exam
The fix is to turn a closed-book exam into an open-book one: give the model the relevant pages at the moment it answers, and ask it to answer from those pages and cite them. That is the idea behind RAG, first published by Facebook AI researchers in 2020. Knowledge lives in a searchable index outside the model, where it can be updated without retraining.
In public benchmarks, giving a model the right documents lifts accuracy substantially, yet even the best systems still miss a meaningful share of questions. External knowledge is the difference between a model that guesses and one that grounds its answers. How you deliver that knowledge, and how you govern it, decides whether the result is enterprisegrade.
2. The three approaches, explained simply
All three approaches answer the same question — how does the right knowledge reach the model? — but they do it at different times and in different forms: RAG fetches it when asked, CAG prepares it in advance, and KAG organizes it into connections first.
An everyday analogy
Imagine three ways to help an expert answer customer questions:
RAG is the research librarian. Every time a question arrives, the librarian searches a huge, constantly updated library, pulls the three or four most relevant pages, and hands them to the expert. It copes with very large libraries and stays current as long as new and changed documents are indexed promptly. But if the librarian grabs the wrong pages, the expert answers from the wrong pages.
CAG is the open binder on the desk. For a small, stable reference — say a 150-page product handbook — the expert reads it once in the morning and keeps it open. Every question that day is answered without fetching anything. Fast and simple, but only works if the binder is small, does not change during the day, and everyone asking is allowed to see all of it.
KAG is the investigator's evidence board. Before any questions arrive, an analyst pins every person, company, contract and event to a board and draws the connections between them, each string labelled and tied to a source document. Questions like "which suppliers of Plant A have contracts expiring within 90 days?" are answered by following the strings. Powerful for connected questions, but building and maintaining the board is real work.
Figure 2 · the three approaches as everyday analogies
Read each column top to bottom: the difference is when the knowledge work happens — at question time (RAG), once in advance (CAG), or as a map built in advance and walked per question (KAG). The same colours mark the three approaches in every diagram that follows.
The same idea in one table
| RAG | CAG | KAG | |
|---|---|---|---|
| In one phrase | Find what is relevant | Reuse what is prepared | Connect facts, then reason |
| When knowledge is selected | At question time | Ahead of time, once | Ahead of time (the graph), then planned at question time |
| Form the knowledge takes | Retrieved text passages | The model's saved "reading" of the whole corpus (KV cache) | Entities, relationships and rules, linked back to text |
| Best for | Large, changing, permissioned libraries | Small, stable, shared references with heavy reuse | Multi-step, relational, time-bound or rule- bound questions |
| Main business risk | Wrong or missing answers when retrieval misses | Stale or over-shared content | High build and upkeep cost; wrong links in the graph |
| Typical owner | Platform or search team | Platform team, with content owners | Data and domain teams together |
3. A better mental model: selection, reuse and structure
Instead of asking "RAG, CAG or KAG?", ask three independent questions of every request — what should be selected, what can be reused, and what must be structured — because a single answer can need all three.
Three design axes
| Axis | The question it answers | Primary technique | Typical failure if ignored |
|---|---|---|---|
| Selection | Which information is relevant_and authorized_ for this request? | Retrieval: keyword, semantic, SQL, API, graph lookup (RAG) | The model answers from the wrong or forbidden material |
| Reuse | Which work can be shared across requests without breaking freshness, permissions or model compatibility? | Caching: prompt/prefix and KV caches (CAG) | Paying again and again to re-read the same handbook |
| Structure | Which entities, relationships, time limits, rules and calculations must be explicit rather than buried in prose? | Knowledge graphs, logical forms, deterministic operations (KAG) | Plausible but wrong joins, dates and arithmetic |
Seen this way, hybrids are natural rather than exotic. A cached handbook does not stop the system retrieving a newly issued notice. A graph traversal does not stop it retrieving the supporting clause. A retrieval pipeline can still cache its stable instructions.
Four kinds of enterprise information
The three axes become concrete once you sort enterprise information by what it is. Most knowledge fits one of four categories, and each has a natural home:
| Category | Examples | Preferred treatment |
|---|---|---|
| Unstructured evidence | Contracts, policies, manuals, tickets, reports | Retrieval (RAG), including multimodal retrieval for drawings and scanned pages |
| Stable shared context | System instructions, approved reference sets, a product handbook | Prefix caching, or bounded- corpus CAG where it tests well |
| Relational knowledge | Suppliers and the parts they supply, asset hierarchies, dependencies, ownership | Graph queries (GraphRAG / KAG) or relational joins |
| Authoritative operational state | Inventory, balances, open orders, production events, credit limits | Governed APIs, SQL and semantic-layer queries against the system of record |
The fourth category is the one most often mishandled. A vector index or a knowledge graph should never become an accidental copy of transactional data: the copy goes stale within minutes, and it bypasses the authorization rules of the system that owns the data.
Part II — How each approach works
4. Inside RAG: dynamic evidence selection
RAG is the enterprise default for grounding models in large, changing, permissioned knowledge. In my experience its quality is set far more by retrieval engineering (how documents are parsed, chunked, searched and reranked) than by the choice of vector database, which is where too many programs spend their first procurement cycle.
The pipeline at a glance
Every RAG system has two halves. An offline half prepares the library: it parses documents, splits them into chunks, converts each chunk into numbers that capture its meaning (embeddings) and stores them in search indexes. An online half answers questions: it searches those indexes, picks the best evidence, and asks the model to answer from it.
Figure 3 · RAG reference pipeline: offline ingestion and online serving
The top lane runs whenever documents change; the bottom lane runs for every question and snakes from the question to a cited answer. The dashed line is the only link between them: the search step looks up the indexes the offline lane built.
The four levers that move quality
The best public evidence that chunk context, hybrid search and reranking compound comes from Anthropic's 2024 contextual retrieval study. Adding chunk context, then keyword search, then reranking each reduced how often the right passage went missing, and together they cut those failures by roughly two thirds. The one-off cost of adding context was trivial with prompt caching. These were the vendor's own datasets, so test on yours, but the direction matches what I see in client pipelines: most RAG quality problems are retrieval problems, and they are cheap to fix.
5. Inside CAG: reusing a stable knowledge prefix
I treat CAG as a deployment pattern rather than an architecture: if a small, stable body of knowledge is going into the prompt anyway, compute the model's reading of it once and reuse it. Every major model vendor now offers this as a managed feature. Its practical ceiling is set by how much context a model can actually use well, not by the window size on the datasheet.
How CAG works
The pattern was named in a 2024 paper with the deliberately provocative title "Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks" (Chan et al.). It describes three phases:
Preload. Fit the whole document collection into the context window, run the prefill once, and store the KV cache.
Infer. For each question, load the cache, append the question, and generate. No retrieval runs.
Reset. Truncate the question tokens from the cache so it returns to the clean knowledge state.
- "Reset" only clears the last question. It does nothing about corpus updates: any change to the knowledge means recomputing the cache.
Figure 4 · CAG reference flow: preload once, serve many
The top lane is paid once per corpus version; the bottom lane runs per question. The loop on the right is the reset that returns the cache to its clean state. The only way new knowledge enters is to rebuild the top lane.
What the original study really shows
The study is widely quoted and rarely read. It used one small open model, small corpora and a soft similarity metric. CAG edged out RAG on most of the small tests by very thin margins, and lost on the largest corpus, the one closest to a real knowledge base. The authors themselves call CAG impractical for significantly larger datasets.
The famous "40x faster" compares CAG with re-reading the whole corpus on every question, not with RAG. A simple keyword RAG pipeline was faster than CAG at every size tested, and CAG's response time grows with the corpus while RAG's stays flat. What the paper proves is that if a small corpus goes into the prompt, it should be cached. It does not prove that caching beats retrieval at enterprise scale.
Where CAG fits
My guidance: consider bounded-corpus CAG when the authorized content fits comfortably within the context the model uses well, leaves room for the question, history and answer, and holds up on quality, latency and cost in a side-by-side test against tuned RAG. Good candidates change rarely, are queried heavily, and need no per-user permissions inside the corpus: HR and benefits handbooks, product FAQs, support playbooks, standard operating procedures, a single contract or regulation, and agent system prompts with tool documentation. Poor fits are large or fast-changing catalogues, multi-tenant document sets, permissioned content, and audit regimes that require citing the exact passage used.
6. Inside KAG: structured knowledge and executable reasoning
Graph methods pay off on questions that chain entities, depend on dates, apply rules or need a view across a whole portfolio, and they lose to plain RAG on simple lookups. For a technology leader the more important point is that which graph method you choose matters more than whether you use a graph at all: the cost of answering a question varies by orders of magnitude across methods.
Why similarity search is not enough
Consider: "Which supplier-related events contributed to production losses at Plant A last quarter, and which purchase orders were affected?" Answering it means connecting a supplier → a delivered component → an equipment asset → a maintenance order → a production interruption → a financial impact → purchase orders. Those links live across
SAP, maintenance records, contracts and incident reports. A vector retriever finds documents that sound related; it does not establish the relationships between the underlying business entities. That gap is what KAG targets.
The KAG framework (Ant Group / OpenSPG)
KAG comes from Ant Group, which published it in 2024 and open-sourced it as OpenSPG/KAG. It targets two RAG weaknesses: the gap between "sounds similar" and "is relevant", and blindness to "knowledge logic" such as numbers, time relations and expert rules. It combines a domain schema, two-way links between graph and source text, and a planner that turns each question into executable steps.
In short: KAG is GraphRAG plus a symbolic planner-executor plus a domain schema. Its distinctive contribution is the logical form — an auditable, step-by-step plan that can include arithmetic and deduction.
Figure 5 · KAG reference flow: builder and solver
The builder runs offline and keeps the graph and the text chunks linked both ways. The solver turns each question into a plan, executes it against graph, text and tools, and loops back to re-plan when evidence is missing.
What the KAG results mean for an enterprise
On academic multi-hop benchmarks KAG posted the largest gains of any graph method, though those figures come from its authors and are often quoted wrongly. The more persuasive evidence is from Ant Group's own production use: in government-services Q&A, precision rose from roughly two thirds with naive RAG to above 90%, with similar improvements in health-insurance policy questions. The authors are candid about the cost: many model calls to build the graph and to plan each answer, planning that roughly doubles output, slow entity standardization, and unresolved extraction consistency. KAG buys precision on rule-bound domains with engineering effort; it is not a free upgrade.
The open-source framework is maturing. Recent releases cut graph-construction cost substantially and added MCP support, which positions KAG as a knowledge service that agents can call rather than a standalone application. Domain variants are appearing in areas such as building-engineering standards, which is exactly where I expect KAG to earn its keep: dense, rule-heavy reference material where a wrong clause is expensive.
Independent evidence: graphs help on relations, hurt on lookups
Two independent studies published in 2025 and 2026 compared graph methods with plain RAG on the same questions. Their findings line up, and they are what I use to set expectations with business sponsors:
| Question type | Better choice |
|---|---|
| Simple fact lookup | Plain RAG, slightly |
| Detail questions on a long document | Plain RAG, clearly |
| Multi-hop questions | Graph methods, slightly |
| Time-dependent questions | Graph methods, by a wide margin |
| Themes and summaries across a corpus | Graph methods, clearly |
| Questions with no answer in the data | Plain RAG: graph methods often answer anyway instead of abstaining |
Three details matter for planning. Automatically built graphs captured only around two thirds of the entities needed to answer the questions. Each approach answered a meaningful set of questions the other missed, and combining both beat either alone. And model-based judges flipped their verdicts when two answers swapped position, which undermines many published "win rate" claims. The case for graphs is real but narrow, and it is strongest in a hybrid.
Part III — Comparing and combining
7. Head to head: what the evidence means for your architecture
No approach wins everywhere, and nobody has yet compared all three on the same data. What the independent studies do agree on is enough to decide: good retrieval matches long-context prompting at a fraction of the tokens, graphs win on multi-step, time-based and summary questions but lose on simple lookups, and combining approaches beats any single one. For a CTO, the more important differences are in cost.
Token usage: where the tokens go, and how to govern them
When I review the bill for a production assistant, the spend is rarely where the team expects. It is driven less by the answer than by everything sent alongside the question: instructions, conversation history, retrieved evidence, cached content, tool results and, in agentic systems, every intermediate step. Each approach shapes that mix differently.
Figure 6 · where the tokens go in one request, by approach (illustrative)
The numbers are illustrative, not measured, but the shapes are typical. RAG keeps each request small by sending only a few reranked passages. CAG sends the whole corpus every time; caching makes that affordable, but only while the cache stays warm and the corpus stays small. Graph approaches add planning calls and intermediate results. Agents are the real multiplier: each step re-sends the instructions and the growing context, so cost grows much faster than the number of steps unless context is pruned between them.
| Token component | What drives it | How to control it |
|---|---|---|
| Instructions and tool definitions | Prompt length; agents carrying every tool's schema on every call | Cache them; give each route only the tools it needs; load tool definitions just in time |
| Conversation history | Long sessions replayed in full | Summarize or truncate; keep durable facts in a memory store, not the transcript |
| Retrieved evidence (RAG) | Number and size of passages | Rerank to fewer, better passages; adapt how many to the question; compress |
| Cached prefix (CAG) | Corpus size times question volume | Billed at the cached-read rate; watch rewrites on expiry; keep volatile content out of the prefix |
| Graph results and plans (KAG) | Traversal breadth, community summaries, planning steps | Prefer local over global search; cap hops; return paths and IDs, not whole documents |
| Tool outputs and intermediate steps | Agent iterations; verbose tool responses | Step and token budgets per request; trimmed, structured tool outputs |
| Reasoning tokens | Model "thinking" settings | Set reasoning effort per route; reserve deep reasoning for hard question classes |
| Output | Answer length and format | A clear answer contract; concise formats; cite by source ID |
Treat tokens as a managed resource, not a by-product. In practice that means:
Budgets. Set a token budget per use case and a hard cap per request, enforced by the orchestrator, with a defined behaviour when it runs out (answer what is supported, list the gaps).
Routing to the right model. Send simple, well-grounded questions to smaller, cheaper models and reserve frontier models for the question classes that need them. Output tokens typically cost several times more than input tokens, so verbose answers and heavy reasoning are disproportionately expensive.
Metrics. Track input versus output tokens, cache hit rate, tokens per request and per correct answer, and steps per agent task, by use case. Alert on runaway agent loops.
Showback. Report token spend by business unit and use case next to the value it produced. Cost per correct, authorized answer is the number that keeps the conversation with the CFO grounded.
8. Combining them: hybrid architecture and context engineering
The research groups behind all three approaches have each moved towards hybrids, and the 2026 practice of context engineering treats RAG, CAG and KAG as complementary services that an orchestrator combines to assemble the smallest sufficient evidence for each step. This is where I would point any enterprise architecture.
How the semantic layer should operate
Most programs I review treat the semantic layer as a BI artefact. In an AI architecture it is something more important: the contract between the business's numbers and the model. The model never calculates a governed metric itself and never queries raw tables. It asks the semantic layer, which returns the number together with its definition and provenance.
Figure 7 · how a governed semantic layer answers a numeric question for an AI assistant
The model's job is limited to steps 1, 2 and 7: understanding the question, mapping it to a governed metric, and explaining the result. Everything that must be exact (joins, formulas, currency conversion, fiscal calendars, access rules) happens deterministically inside the layer. That division is what makes numeric answers auditable.
The operating principles I recommend:
One definition per metric, owned and versioned. "Downtime cost", "on-time delivery" and "active customer" each have a named business owner, a written definition and a version. When the definition changes, old versions stay queryable for audit.
Expose a service, not a schema. Publish metrics through a governed API or MCP tool with typed parameters (metric, entity, period, filters). The model fills in the parameters; the layer writes the query. Free-form SQL generated against raw ERP tables is the single most common source of confidently wrong numbers.
Enforce access under the user's identity, inside the query. Row- and column-level rules come from the same policy engine as the rest of the platform, so a user sees the same numbers in the assistant as in their reports, and no more.
Return metadata with every number. Definition version, as-of timestamp, source system and the query used travel with the result, and the answer cites them.
Declare freshness per metric. Some numbers are read live from the ERP, others refresh hourly or daily from the warehouse. The answer should say which.
Share identities with the graph. The semantic layer and the knowledge graph use the same entity and master-data IDs. The graph answers "which suppliers affect plant 1000"; the semantic layer answers "what that cost"; the IDs make the two answers line up.
Separate certified from exploratory. Only certified metrics may feed answers that go to customers, regulators or the board. Exploratory queries are allowed but clearly labelled.
Put metric changes through the release gate. Definition changes re-run the evaluation set, including a check that natural-language questions still resolve to the right metric.
Cache results safely. Key cached results by metric version, user scope and as-of time, and invalidate them on data refresh.
| Responsibility | Owner |
|---|---|
| Metric definitions and certification | Business data owners, typically finance and operations |
| Semantic model, API and MCP tooling | Data platform team |
| Access policies | Security and data governance |
| Question-to-metric resolution quality | AI platform team, through the evaluation set |
In SAP landscapes much of this layer already exists. CDS views and the virtual data model in S/4HANA, and the analytical models in SAP Datasphere and Business Data Cloud, carry SAP's business semantics and authorizations. AI should consume these rather than rebuild definitions. Where a metric spans SAP and non-SAP systems, define it once in the enterprise semantic layer and expose it to agents as a governed tool.
Reference architecture: a four-tier enterprise knowledge platform
This is the target architecture I recommend: a cached core, hybrid retrieval, a selective graph layer and a semantic layer, with a router in front and governance across every path. It is not a vendor standard. Its main advantage is that it turns an irreversible architecture bet into a reversible routing decision.
Figure 8 · four-tier enterprise knowledge platform
9. Decision framework
My default for every new use case: start with a hybrid retrieval baseline unless the whole corpus is small, stable and shared. Then add a cached core, a graph layer and a router only when the workload proves each one pays for itself. This keeps early investment small and makes every later addition a measured decision rather than a bet.
The decision tree
Figure 9 · decision tree, from use case to architecture
Only the first question can end the tree early. After the retrieval baseline, each "yes" adds a layer and the flow continues, so a mature system often ends up with all four tiers of Section 8, each justified by its own evidence.
Recommended starting routes for the router: stable handbook questions use the cached core; anything mentioning recent amendments or dates uses retrieval; exact numeric aggregation calls deterministic computation (SQL or a calculator), never the model's arithmetic; entity-chain questions use the graph; corpus-wide "themes" questions use summaries or GraphRAG-style global search; weak evidence triggers a targeted search or an honest "insufficient evidence" — not unlimited autonomous retries.
Part IV — Running it in the enterprise
10. Security, governance and trust
In my experience, the failures that reach the board are rarely about model capability or retrieval accuracy. They come from weak access control, poisoned or stale sources, and uncontrolled agent actions, and each architecture fails in its own way. This is the section I would ask every CISO and data-protection officer to read.
Rule one: authorize before knowledge reaches the model
If an employee may read a plant's maintenance records but not confidential supplier pricing, the pricing must never enter that employee's model context. Filtering the final answer is not enough; unauthorized data must not be supplied in the first place. The same applies to relationships: a user allowed to see two entities is not automatically allowed to discover a confidential relationship between them, and a derived relationship or summary can reveal what no single visible document does.
This is where the three architectures differ most:
RAG and graph retrieval can filter by permission at query time, so each user's context holds only what they are entitled to see.
CAG preloads the whole corpus, so every user of a cache sees all of it. It suits only content everyone may see, or needs a separate cache per access tier — multiplying write costs and memory.
Graphs must propagate permissions from source chunks to the entities, edges and community summaries derived from them.
Figure 10 · three enforcement stages, one policy decision point
Stage 1 is where RAG, CAG and KAG differ most. Retrieval can filter per user inside the search. A graph has to filter nodes, edges and any summaries built from them. A cache cannot be filtered at all once it is built, so the only option is a separate cache per access tier.
Deletion and revocation must reach every derived artifact
When a document is deleted or a user loses access, the change has to propagate to everything built from it: index entries, embeddings, cached retrieval results, KV caches and prefix caches that contain it, graph nodes and edges extracted from it, and community summaries that mention it. Rebuilding all of that takes time, so the order matters. Serving policy changes first and immediately, so requests stop using the revoked knowledge even while the rebuild runs:
Access change recorded (deletion, revocation, legal hold release)
Serving-policy denial takes effect at once for the affected scope
Affected artifacts retired from serving: index entries, caches, graph elements and summaries are marked unusable
Rebuild the affected index segments, caches and graph regions
Validate that the rebuilt artifacts no longer contain the removed content
Publish a new, version-consistent manifest across index, cache and graph
The gap between steps 2 and 6 is where most leakage happens in practice, which is why revocation propagation time belongs on the evaluation dashboard (Section 11).
11. Evaluation and benchmarking
Compare workloads, not labels. An architecture that wins on an academic benchmark may lose on your SAP maintenance data, and a cache that cuts response time may not improve a single answer. Every organization needs its own evaluation set, run the same way across every option. I treat this as the single most important investment in the program, because it is what lets every later decision be made on evidence.
A minimum viable evaluation for the first 15 days
Teams often postpone evaluation until it is too late to change the architecture. This is a starting kit that fits inside the first phase of the roadmap:
| Element | Minimum setup |
|---|---|
| Question set | 200–300 questions reviewed by a domain expert, with about a quarter held back for final testing |
| Question mix (starting point) | About 30% single-hop factual, 20% multi-hop, 10% time-based, 10% numeric or rule-based, 10% summary, 10% unanswerable (should abstain), 10% permission-boundary cases |
| Core quality metrics | Groundedness, answer relevance, citation precision, correct abstention |
| Core operational metrics | p95 latency; cost per correct, authorized answer |
| Diagnostic | An oracle-evidence test: give the model hand- picked evidence to separate retrieval failures from generation failures |
| Judging | A model as judge with randomized answer order, plus a human-reviewed sample, because judges favour position and length |
| Statistics | Paired comparisons on the same questions, with confidence intervals, reported per question class |
| Regression gate | Every change to prompts, models, chunking, indexes, caches or graphs re-runs the set; a drop beyond an agreed margin blocks release |
Add adversarial tests early: injected instructions inside documents, poisoned nearduplicates, questions that try to elicit restricted relationships, and stale-version traps.
12. Industry use cases
Asset-intensive and regulated industries get the most from combining all three approaches, because they already keep an implicit ontology (asset registers, product hierarchies, policy rules) and ask genuinely multi-hop questions — but the authoritative numbers must always come from systems of record, not from the graph or the model.
Architecture by industry at a glance
| Industry | CAG (cached core) | RAG (hybrid retrieval) | KAG / GraphRAG (graph layer) | Where I would start |
|---|---|---|---|---|
| Energy / oil & gas | HSE procedures, permit-to- work rules, SOPs | Well reports, drilling and maintenance logs, engineering documents (often visually rich) | Field, facility, equipment and component hierarchies; failure modes; procedure links | Multimodal RAG over engineering documents, then an asset graph |
| Manufacturing | Product-line FAQs, safety procedures | Manuals, service bulletins | Bill-of-materials and asset/failure graphs | RAG for service, graph for root-cause and supplier impact |
| Healthcare / life sciences | Clinical protocols, formularies, benefit summaries | Literature, payer policies, clinical notes (strict access control) | Drug–gene– disease graphs, claim-to-policy relationships | Permission- aware RAG; KAG for payer- policy rules |
| Financial services / insurance | Product and policy FAQs | Research, filings, contracts | KYC/AML entity networks, riders and exclusions, regulatory lineage | RAG for research; graph for entity risk and policy rules |
| Public sector | Citizen- service FAQs | Permission- aware search with data residency | Rule-bound eligibility | Cached FAQs plus rules- driven eligibility |
| Legal | One contract or regulation under review | Contract and clause search | Clause and precedent citation graphs | RAG with exact-passage citation |
| Retail / customer support | Returns, warranty and shipping policies | Orders, products and inventory via retrieval and live tools | Supplier and part networks | Cached policies plus live order lookup |
| IT operations / cybersecurity | Runbooks | Logs and tickets (just-in- time search suits logs) | Configuration- item, change and incident dependency graphs | Cached runbooks plus a dependency graph |
SAP and ERP landscapes
ERP systems are the clearest case for combining all three approaches, because they hold transactional truth that changes by the minute, business semantics encoded in the data model, and fine-grained authorizations — none of which a vector index or a cached prefix can safely replace. The rule of thumb for SAP landscapes: documents go to RAG, stable reference goes to the cached core, relationships go to the graph, and live facts come from SAP itself.
SAP's own building blocks map onto the three approaches
SAP's AI stack now contains a native counterpart for each layer (status as of November 2026; check exact availability for your edition and region):
| Approach | SAP building block | What it does |
|---|---|---|
| RAG | SAP HANA Cloud vector engine and document grounding in the generative AI hub and Joule | Generally available since 2024; the document-grounding module implements RAG on top of it |
| KAG / graph | SAP HANA Cloud knowledge graph engine and SAP Knowledge Graph | Generally available since 2025; SAP positions SAP Knowledge Graph as the semantic bridge between Joule agents and business data and processes |
| Unified data | SAP Business Data Cloud (with Databricks) | The trusted data layer that grounds Joule agents across SAP and non- SAP sources |
| Agents and routing | Joule, Joule Agents and Joule Studio | Custom agents grounded on business documents and data, with MCP and agent-to-agent collaboration, and governed execution of multi-step and third- party agents |
| CAG | No SAP-branded equivalent | Prompt caching depends on the mode...Source:https://dev.to/ganesh_raju_6a2557b4199e6/rag-cag-and-kag-choosing-how-enterprise-ai-uses-what-the-business-knows-3d1m PreviousI Hired Five Agents. Now I Work for Them.NextRedis SCAN Returned 1,973 Empty Pages and Found All 20 Keys Related posts![]() How to Check Whether an AI Image Edit Kept the Composition0dev.to Mac and Electron app protection tools compared0dev.to Stop Making Your AI Carry the Data. Let It Write the Program.0dev.to Cron Backups of App Data to an S3-Compatible Bucket (Presigned Restore Included)0dev.to Two Agents, One Machine: When Multiple AI Agents Collide0dev.to Before You Sign an IT Support Contract in Oman: A 10-Point Checklist0dev.to show = false, 2600)"
x-show="show" x-cloak x-transition.opacity.duration.200ms
class="fixed bottom-8 left-1/2 z-[80] -translate-x-1/2 rounded-lg border border-line bg-card px-4 py-2.5 text-sm text-heading shadow-card-hover"
role="status" aria-live="polite">
|










