The biggest business intelligence vendors now all sell an AI agent. Microsoft has Copilot in Power BI and build-your-own Fabric Data Agents. Salesforce rebuilt Tableau around agents grounded in a semantics layer. ThoughtSpot replaced its conversational interface, Sage, with an explicitly agentic analyst called Spotter. Sigma and Hex have agents of their own. The industry’s bet is unmistakable: the next interface to enterprise data is not a better dashboard but a system that plans multi-step analyses, writes and runs code, checks its own work, and hands back findings.
I spent months reading the research literature behind that bet — 68 studies and systems, from text-to-SQL methods to failure taxonomies — and I came away with an uncomfortable conclusion. The products are real. The capability claims are not. And the gap between the two is not mainly a model problem. It is a measurement problem, a semantics problem, and a trust problem, in that order.
The benchmarks nobody puts in the sales deck
When researchers test agents on realistic analytics work instead of isolated queries, the numbers collapse. Spider 2.0, a benchmark of 632 authentic enterprise data-analysis workflows drawn from professional settings, finds the best agent framework in its paper completing roughly 21 percent of them. DABStep, built from more than 450 multi-step tasks taken from a real financial analytics platform, is harsher: the best agent in its paper solves only 14.55 percent of its hard tasks. These are not toy failures. DABStep is deliberately adversarial to the field’s habits — it scores factoid answers with automatic correctness checks, no LLM judge in the loop, so a fluent answer cannot talk its way into a pass.
Multi-agent systems, the architecture behind most of the marketing, fare no better. The MAST study built an empirically grounded failure taxonomy from 1,642 expert-annotated traces and reports failure rates between 41 and 86.7 percent. Crucially, the failures concentrate in specification breakdowns — agents misunderstanding what was asked — and in coordination breakdowns and weak task verification. They do not concentrate in “the model was too small.” Adding agents multiplies coordination failures and token costs faster than it adds capability. Anyone evaluating an analytics agent should ask for its numbers on workflow benchmarks like these, not on single-query accuracy, because single-query accuracy is close to meaningless as a predictor of production behavior.
The architectures underneath these systems reduce to four recurring patterns: planner–executor loops with an optional critic, role-specialized pipelines that pass structured artifacts between fixed roles, graph-orchestrated swarms, and search-augmented planners that explore multiple solution paths. No pattern dominates; the right choice depends on the shape of the task. What the best-evaluated systems share is not agent count but discipline — explicit coordination and aggressive verification of intermediate results.
The agent does not know what “revenue” means
Here is the part of enterprise analytics that agent research mostly ignores. In a mature BI practice, analysts almost never query raw tables. They query a semantic layer: a governed mapping between business concepts and physical data, with certified definitions, hierarchies, and access rules. “Revenue” has exactly one authoritative definition — its grain, its filters, its certification status. When two dashboards disagree on revenue, the result is not a rounding issue; it is a trust-destroying event that takes quarters to repair.
An agent that bypasses that layer and writes SQL against raw tables, or quietly invents a plausible definition of a KPI, produces output that looks authoritative and is wrong in exactly the way enterprises fear most. The serious vendors understand this. Tableau grounds its agents in Tableau Semantics, and Microsoft’s Fabric data agents are built to answer from Power BI semantic models — governed definitions, not raw tables — and the orchestration-layer thinking in products like Domo’s agentic stack points the same way: governance is the product, not the garnish. A fellow HackerNoon contributor put the adjacent point well — an agent is only as smart as the data pipeline beneath it.
What is missing is narrower than “no standard.” The Open Semantic Interchange — launched by Snowflake, Salesforce, dbt Labs, and others in 2025, finalized as a specification in January 2026, and now incubating at the Apache Software Foundation as Ossie — standardizes how metric definitions are exchanged between tools. What it does not yet standardize is the agent side of the contract: how an agent consumes a certified definition at query time, proves that a reported number actually came from it, or behaves when two governed definitions conflict. Most deployments still wire that grounding up by hand, platform by platform. Until that contract exists between the governed layer and the agent, benchmark progress and production readiness will remain two different things.
Knowing the privacy rule is not the same as following it
The trust evidence is worse than the accuracy evidence. Rahman and colleagues, surveying 45 data-science agents, found that over 90 percent lack explicit trust and safety mechanisms. ToolEmu, a risk-evaluation framework covering 144 high-stakes tool-use cases, found that even the safest agent it tested fails 23.9 percent of cases, including private-data leaks and financial-loss scenarios.
The most instructive result is PrivacyLens. Across 493 scenarios, models answered questions about privacy appropriateness with roughly 90 to 95 percent accuracy — and then leaked the same sensitive data in 25 to 40 percent of their agent actions. Declarative knowledge does not transfer to agentic behavior. An analytics agent can recite the access policy and still break it, which is why privacy has to be enforced by the system around the agent — row-level access controls applied at runtime — rather than requested of the agent in a prompt. The same logic applies to security: an analytics agent’s job is to ingest untrusted spreadsheets, logs, and documents, which makes prompt injection in its indirect form — prompt injection is the top entry, LLM01, on the OWASP Top 10 for LLM Applications — a design condition rather than an edge case.
We can grade answers. We cannot grade insight.
Evaluation in this field has moved along a single axis: correctness, then efficiency, then realism. Spider graded exact-match correctness. BIRD added an efficiency score, its Valid Efficiency Score. Spider 2.0 and DABStep grade realistic workflow completion. The next step on that axis is the one that actually matters to a decision-maker — insightfulness: whether a finding is novel, decision-relevant, and worth acting on.
For that, no validated metric exists. The closest attempt, InsightBench, plants known insights in 100 business datasets and grades whether the agent recovers them. Recovering a planted insight is a useful proxy, but it is not the same as producing a good one, and grading insight with an LLM judge imports the biases of the very class of systems being evaluated. So leaderboards keep rewarding what is measurable — query correctness, task completion — while the properties that decide production success go unmeasured: did the agent use the certified metric definition, is the finding actually new, would a decision-maker trust it?
Insightfulness is not unmeasurable in principle. A credible harness would be hybrid: deterministic checks for what can be checked (every reported number recomputable from the data and the governed definitions, required analysis steps actually performed, certified definitions actually used), and bias-controlled human judgment for the rest — blinded judges, reported inter-rater agreement, calibration against gold-standard analyses. None of that exists as a validated instrument today. Until it does, no one can distinguish, at scale, a fluent-looking analysis from a genuinely insightful one.
What I would demand before letting an agent near a decision
None of this argues against analytics agents. It argues for buying and building them honestly:
- Workflow benchmarks, not query benchmarks. Demand Spider 2.0- and DABStep-style completion rates for systems that claim to do end-to-end analysis.
- Governed semantics as a hard requirement. If the agent cannot consume certified metric definitions and show lineage for every number, it is a demo.
- Runtime enforcement, not prompt-level policy. Privacy and access rules belong in the system around the agent, because the evidence says the agent will not reliably enforce them itself.
- Governance frameworks applied, not cited. The NIST AI Risk Management Framework and the OWASP Top 10 already give security and compliance teams the vocabulary; the missing habit is applying them to analytics agents specifically.
I have written up the full evidence base — the architecture taxonomy, the benchmark record, the deployment analysis, and an eight-item research agenda — in a survey, Agentic AI Meets BI: Architectures, Evaluation, and Deployment for Analytics Agents, currently under review at IEEE IT Professional. The short version fits in one sentence: the industry has decided what it wants, the benchmarks say it does not reliably work yet, and closing that gap is now a problem of semantics, measurement, and trust — not of model size.

