Why semantic layers will be essential for enterprise AI

AI can find your data. It can’t read your mind. AI agents can already reach into your CRM, your warehouse, and your ERP, and with standards like MCP, that list keeps growing. Access is solved. Understanding is not. An agent can retrieve every record in your pipeline and still give your CRO the wrong answer, not because the data is bad, but because it doesn’t know what you mean by “pipeline.” Agents without semantics don’t fail loudly. They deliver confident answers to the wrong question.

“Pipeline” means something different at every company

Ask an executive, “How is pipeline looking this quarter?” and it sounds like a simple request. It isn’t. The answer depends on your sales stages, your forecasting methodology, your territory structure, and your qualification criteria, which are business rules that evolved over years, often without ever being written down. Two companies can use the same word and mean entirely different things. Businesses don’t run on raw data. They run on shared meaning, and humans carry that meaning around in their heads. Agents don’t have heads. They have whatever we give them.

From systems of record to systems of agency

That matters more as the way we work changes. For decades, employees logged into the CRM, the BI tool, or the ERP and supplied the judgment themselves, because they knew which dashboard to trust and which field was never quite right. Now they state intent, and an agent figures out the rest. 

The systems of record stay where they are, but a new layer sits between people and technology: a system of agency. In my experience, leaders spend a lot of time asking whether that layer can connect to everything and far less time asking whether it understands anything. If that layer doesn’t understand your business, nothing underneath it matters.

Catalogs helped people find data. Semantic layers help AI understand it.

For the last decade, we invested in governance, integration, quality, and catalogs, and those investments still matter because trusted data is the prerequisite. But a catalog tells people where the data lives. A semantic layer tells AI what it means. It captures what your best people carry in their heads:

• Business definitions, such as what counts as “pipeline,” “churn,” or “active customer”
• Policies that govern how data can be used
• Relationships between products, customers, contracts, and outcomes
• Rules that determine how decisions get made

“A catalog tells people where the data lives. A semantic layer tells AI what it means.”

Put that in a form both humans and machines can consume, and agents stop guessing. This isn’t a nice-to-have. It’s the difference between an agent that is technically capable and one that is operationally reliable.

Relationships are where the value lives

Definitions are only half the story, because business decisions rarely live in a single dataset. They depend on how concepts connect. Take an AI-powered quoting process, where the agent needs products, contracts, customer history, usage patterns, approval structures, and pricing rules all at once, and each lives in a different system. Retrieving them is the easy part. 

Reasoning across them is the hard part, and when the agent gets it wrong, it won’t throw an error. It will produce a plausible quote with the wrong discount and send it to a real customer. That’s where knowledge graphs and context graphs earn their place. They treat information as connected business knowledge rather than isolated facts, so instead of piecing together a customer’s story from scattered records, the agent gets a map of how your business fits together.

Meaning is also an efficiency play

Without encoded context, agents spend compute rediscovering relationships and re-inferring meaning your organization already understands, so every query becomes a small act of reconstruction. Define it once, and every agent reasons from the same source. That means more consistent answers, fewer contradictions between teams, and lower cost as usage scales. Consistency is underrated. If sales, finance, and operations each get a different answer to the same question from the same AI, you haven’t built intelligence. You’ve built an argument.

Anyone can build now. Not everything should scale.

Generative AI has finally delivered on the citizen developer promise. Employees can describe a workflow in plain language and get a working prototype back, which means good ideas can now come from anywhere. That’s the opportunity. 

“If sales, finance, and operations each get a different answer to the same question from the same AI, you haven’t built intelligence. You’ve built an argument.”

The risk is that a tool that helps one person is a convenience, while the same tool deployed across departments is a liability if no one validated it, governed it, or owns it. Organizations need a path from experiment to enterprise asset. I think of it as a graduation factory: identify the promising prototypes, test them, apply governance, and promote the ones that earn it. Innovation should be easy to start and hard to scale carelessly. 

The same foundations apply here. Only 12% of organizations say their data is AI-ready, and only 31% can tie their AI initiatives to clear KPIs. Semantics and governance are how you close that gap.

Where I’d start

You don’t need a multi-year program to make progress. Start small and prove it:

1. Pick the five terms your executives argue about most, and define them.
2. Assign a business owner to each definition.
3. Expose those definitions to your agents through a governed layer, then measure whether the answers improve.

Meaning, like data, has to be managed. Someone has to own it.

The bottom line

Connecting AI to your data is table stakes. Everyone will do it, so it won’t differentiate you. Trusted data is the prerequisite. Shared meaning is the multiplier. Connect your agents to everything, and you’ve built access. Teach them how your business thinks, and you’ve built an advantage.

The post Why semantic layers will be essential for enterprise AI appeared first on The New Stack.