“Let’s connect our warehouse to Claude.” I welcomed the suggestion with both excitement and trepidation. At that time, I remember being equally in awe of the opportunities it would open as alarmed by everything that could potentially go wrong. Let me give you a free tour of what went through my mind.
The part of me that got excited saw everything this would make possible.
In most organizations, the data team is seen as the one standing between other teams and the insights they need. “Bottleneck” and “gatekeeping” are not an unusual pairing when someone is talking about the data team. “Isn’t that dashboard or metric simple enough? Why do I need to create a ticket that will be thrown into the backlog, anyway?” At last, the data would be at the stakeholders’ fingertips.
I was also excited for the data team. Gone would be the days of doing the menial work of writing yet another ad-hoc query, creating really complex dashboards no one would look at, or pulling out fresh numbers so they could be imported into an Excel sheet.
A real win-win.
On the other hand, I also saw how opening up the gates could be problematic. In my experience, the best measure of success of a data team is whether they can build trust in the metrics they produce. A well-governed data architecture. Music to my ears.
Trust in data is built by being consistent and predictable. Metrics need to be deterministic. You don’t want last quarter’s revenue to jump from one day to the next. You don’t want 20 different dashboards reporting 20 different numbers for the same KPI.
AI agents are, by nature, probabilistic. Give your agent instructions to write a paragraph. The same instructions can generate multiple versions. Which one is the right one? For text, it might not matter, but for data, it does.
The main challenge in the age of agentic analytics is to give your agent the right harness. You need to constrain its behavior and make it as deterministic as possible.
Yet I felt there was so much out there that focused on the cosmetic work and the presentation layer (did someone say dashboards?) while overlooking what’s inside.
That’s what motivated me to create The Honest Agent. I certainly want to reap the benefits of agentic analytics. But I recognize that this is only possible if people can trust the answers the agent gives them.
Do you actually know whether your agent is being honest, or are you blindly trusting the numbers it gives you?
How does the Honest Agent work?
Remember the last time you had to take a test? You were given a set of questions that would be graded against the accepted answers.
This is exactly what The Honest Agent does. As simple as that. You first define the set of questions, ideally business-critical ones (the ones you really don’t want the agent to get wrong), and then provide the accepted answers for them. Then it’s quiz and report time.
If you already use dbt, The Honest Agent will feel very familiar. Think of it as dbt for your agent evals: you go from running your evals to the report in three simple commands.
I will briefly go through the main components here. Check the docs if you want to dig deeper.
Connect to your MCP server
The Honest Agent connects to your analytics MCP server, the same one your users reach through tools like Claude Cowork.
I’ve tested the setup with Snowflake and MotherDuck. It should work with any MCP server you can reach with a URL and a token. If yours connects another way, reach out and I’ll look at adding it.
Choose your testing model
The Honest Agent needs an LLM to talk to your analytics MCP server and to grade the answers. For this, you need an API key from Anthropic or OpenAI, depending on which provider your organization uses. Note that this is an API key, not a Claude or ChatGPT subscription.
When selecting a model, check which ones are used in your organization (e.g. Claude Sonnet 5.5 or Opus 5.5). This gets you closer to the answers people are getting. The Honest Agent also lets you compare results across models (more on that later).
Note that each evaluation run incurs costs, and these vary with the model you select.
Define your evals
Defining your evals is just a matter of writing them down in YAML. Each eval can check two things:
Accuracy: is the agent giving you the right answer?
Provenance: is the agent looking at the right sources?1
Why check both? They point to different issues.
Your agent might ace accuracy and fail provenance. For example, instead of the table with the aggregated and curated numbers, it looks at some intermediate ones. It got to the right answer, but do you want your agent relying on those intermediate tables?
Your agent might use the right sources and still fall short on accuracy. It consulted the right tables as part of its work, but got lost along the way.
In my experience, good evals are the ones that cover the questions your users are constantly asking. Take some time to define them. Dig into your warehouse logs and talk to your stakeholders to find out. You won’t be able to cover all the questions, but you can try to cover the most important ones.
Check the docs for details on how to write evals and all the options around them.
Run your evals
You can run your evals with one simple command:
honest-agent runThe run command also offers options to run only a subset of your evals, in the same style as dbt, e.g. honest-agent run --select tag:smoke. Check the docs to learn more.
Build and serve your report
Call me biased, but the report is what all the evaluations are about. A score can only tell you so much if you can’t dig deeper into the questions that failed, the SQL that was run and the steps your agent took.
Building and serving the report is just two commands away:
honest-agent report
honest-agent serveThe report has three sections:
Overview
In one screen, you can quickly compare your agent’s scores. The heatmap gives you an overview of the different eval runs and helps you to spot drifts; click any cell to open that answer. The Below threshold list shows the evals that didn’t pass, and why they failed.
Model comparison
Want to check how different models compare on your evals? Say you want to see how Sonnet 5.5 scores against Haiku 5.5. Or you want to check how OpenAI models compare to Anthropic’s. This is your page.
The heatmap lets you spot the performance gaps across models. You can also track each model’s scores over time. As your data and setup change, you’ll see which model keeps up, and whether a cheaper one is good enough for the questions your team actually asks.
Result page
Admittedly, this is the page where I spend most of my time. It’s like opening a window into your agent’s decision-making process. You can get there by clicking an eval’s cell in the heatmap, or from the Below threshold list in the Overview.
Besides a closer look at how the agent scored on accuracy and provenance, it helps you understand the causes behind those scores.
This is also where The Honest Agent differs most from general-purpose eval tools, which mostly judge the answer as text. In analytics, the answer is a number that came from a query, so The Honest Agent looks at the query too:
Every SQL query the agent ran is extracted and shown to you, so you can quickly check its logic.
That SQL is parsed to find the tables it actually read, and compared with the sources your team trusts. Getting the right number from the wrong table shows up as a failure, not a pass.
Make it part of your CI/CD pipelines
The evals are only useful if you keep running them. Because The Honest Agent is a CLI, it fits easily into any scheduler you already use to run your data pipelines. If an eval falls below its threshold, the run fails, so you’ll notice.
You can also configure a Slack webhook to get alerted when your evals fall below their thresholds.
Is The Honest Agent your ultimate solution to build trust in your agentic analytics?
As the creator of a tool that has “honest” in its name, I should hold myself to the same standard: No. The Honest Agent won’t be your ultimate solution, but it can certainly help you along the way.
There are two limitations you should be aware of:
Each eval is a single, fresh question. In the real world, users interact with agents in a series of questions and answers, where context builds up and shapes the next reply. Multi-turn evaluation is something I want to explore in coming iterations.
You can only catch errors in the questions you are tracking. Think about the millions of questions your users could ask of your data. It is impossible to foresee and evaluate all of them.
Even with these limitations, The Honest Agent already sheds some light on what used to be a black box. Think about it: if your agent fails to answer a simple question with a fresh context, it is most likely also getting it wrong in longer conversations with your users. And while you can’t catch every error your agent might make, wouldn’t it be great to catch the most important and most frequently asked ones? This is where The Honest Agent can help.
Try it out!
The Honest Agent is open source. You can check the repo on GitHub, and follow the setup guide here. If you’d like to see it in action first, have a look at the demo report.
Note that it’s still in alpha, so there might be some breaking changes while I keep developing it. I’d also be very happy to get your feedback: feel free to reach out in the comments or on LinkedIn.
Provenance only sees SQL. If your agent answers through a semantic layer or a search tool, with no SQL involved, its provenance can’t be checked yet.