It’s 7am and you’re ready to start your morning commute. Your phone lights up with a push notification: “Data agent: your daily subscription metrics.” Subscriptions are up, churn is down, and your coffee just started tasting better. You say to yourself: we’re on a winning streak.
You think back, smiling, to that bygone era when you needed to contact the overwhelmed data analyst, create a ticket, and pull every lever to get it prioritized. Data is now where it needs to be: at the heart of the company, pumping every decision, one Slack message away.
You share the news with finance. Instead of celebration, you’re met with a chill. Those new subscriptions you were so excited about never turned into actual revenue. Right. It was too good to be true. A dream, or technically speaking, a hallucination.
Agentic Analytics: the quest for following the right path
Think about the time you asked your favorite AI tool to build a document or presentation for you. You might need to do some small tweaks, change the tone, and so on. But there were many different ways the task could have been done well.
Data is different. Your task is to land on a single answer (say, the number of subscriptions) and that answer has to be correct.
As Anthropic puts it in their article on enabling self-service analytics with Claude, some domains, like coding, have a much wider solution space (the range of answers that would all be acceptable). In data, the path is narrow. Coding is an open-ended space where creativity pays off, and tests and docs catch wrong answers for you. Analytics is the opposite: there’s often, in their words, “only a single correct answer using a single correct source,” with no deterministic way to prove it’s right.
The main task in this new agentic analytics work is to narrow the path the agent takes and lead it to the right solution. The good news is that most of this can already be built with your current stack, if you’ve been following good data practices. Being ready for the agentic era means doubling down on modeling, documentation, and validation. As Anthropic found, that’s the difference between an agent that’s right 21% of the time and one that’s right 95%.
3 pillars for Agentic Analytics
In the new agentic world, context is king. Context, in my view, is the ecosystem in which the agent operates and makes its decisions. In the data world, two pillars build that context: data modeling and documentation. A third one keeps it honest: validation, making sure the context is correct and stays that way as the business changes.
Data modeling foundations
We’ve all been there. You need to put together a quick report on subscriptions. You go digging and find five different datasets that could work, each giving a slightly different number. So you muster some courage and go straight to the source, trying to connect the tables yourself. You spot a c_id (customer, maybe?) but nothing obvious to join it to. You give up. You go for one and hope for the best.
Agents are no different. Five models all covering the same concept don’t narrow the agent’s path to the right answer, they widen it. And when it isn’t clear how those models connect, the way that c_id had nowhere obvious to join, the agent is left guessing the same way you were.
The bottom line: creating data models for agents is no different from creating them for humans. Think of them as data products. They should be intuitively easy to work with, where dimensional modeling earns its keep. They should reduce ambiguity, so no two models do the same thing. Your organization’s metrics should be codified in a semantic layer, so they resolve to a single definition. And tests and freshness checks keep your sources reliable.
Documentation
Remember that surge in subscriptions that never turned into revenue? Finally your analyst had some time to check it out. The data was “technically” correct, but the surge came from a migration of old accounts to the new subscription provider.
If you’ve worked with data, you know the numbers you see are only part of the truth. The business context and the processes that produce them matter just as much. This kind of knowledge is cumulative, scattered across the organization, and often not written down anywhere.
This is where documentation sets a different bar in the agentic world. It’s not just keeping your dataset descriptions up to date. It means capturing all that process and business logic, the things that usually live in people’s heads, in a written form an agent can actually retrieve and use. Think about the documentation you wish you’d had when you were onboarded.
Your documentation should cover at least three things. First, your data sources: what each one is, what its columns mean, how it relates to others, and what events to account for when interpreting results (remember the subscription migration?). Second, business context: the campaign that drove a spike in registrations, the roadmap change that shifted a metric, the things that explain why the numbers look the way they do. Third, skills.
Skills, as Anthropic describes them, are the procedural knowledge that tells the agent how to navigate everything you’ve documented: which sources to consult first, how to handle ambiguity, and when to hold back rather than answer from a missing or conflicting source. It’s the layer Anthropic credits with the largest gains in accuracy.
Validation
Let’s say you’ve invested the time and effort: proper documentation, business context, sharpened agent skills. Is that it?
That’s only part of the work. Data sources and agent development move fast, and your context goes stale with them. Anthropic watched their own accuracy drift from 95% to 65% in a single month before they started actively maintaining it. So you need to keep your context current and keep checking that your agent’s quality hasn’t degraded over time.
A best practice here is to move context and modeling into the same repository and treat context like any other code change. You can also add a skill that checks, whenever you update a data model, whether the documentation needs updating too.
Another important part is constantly checking your agent’s accuracy. Think of it like quizzing it on the same questions you already know the answers to. Then, when you ship a new model version or edit a skill, you can see whether it actually helped or quietly made things worse.
Can we guarantee the agent will always get it right? Not really. At least, not yet. If you think about it, there’s a catch in that 95% accuracy Anthropic reports, one they acknowledge themselves. You can only measure accuracy on questions you already know the answer to. The silent failures, by definition, are the ones that never make it into your test set. You can’t quiz the agent on a mistake you don’t know it’s making. But that’s why validation is central, and never finished. You can’t eliminate the silent failures, but you can keep shrinking the space where they hide: catching drift, harvesting corrections, sanity-checking the numbers that matter most.
Conclusion
The agentic analytics era is here to stay. As I’ve argued, whether agents succeed in your organization hinges on reducing ambiguity and narrowing their options down to the single right answer.
That demands doubling down on some good old data practices, like building intuitive data models and investing the time to document them. It also means adopting new ones: curating your business context, sharpening your agent skills, and keeping everything honest with continuous validation.
And it demands accepting a trade-off. Democratizing data through agents means there will be cases where someone trusts a wrong answer without ever realizing it. The real challenge ahead isn’t building the agent, it’s making sure that when it’s wrong, you find out before a bad decision gets made.
That world, where your 7am Slack digest steers you right instead of wrong, is a dream well within reach. But it isn’t built by pointing an agent at your warehouse and hoping for the best. What separates dream from hallucination? It’s the unglamorous work underneath: the models, the documentation, the validation. Get it right, and it stays a dream.