But first, let’s start with a dramatization, based on real life, of a company that has just connected Claude to their warehouse (reader discretion is advised).

Clara: “I just updated the investor deck. It was super easy this time! I asked Claude to get the numbers directly from the data. And I’ve been creating these cool dashboards too. Look!”

Three hours later…

Clara: “Just double-checked with the data team. Turns out the numbers were completely wrong. Good thing I hadn’t sent the deck yet.”

Clara was lucky this time, but who knows how many people out there are blindly trusting whatever their agent confidently presents to them as facts. Without a way to verify your agent’s output, you’re in for a wild ride through uncharted territory. Evals help you regain control and visibility over your agent’s actions.

The idea behind evals is simple. There’s no better way to understand how your agent performs than giving it a series of tasks and grading its output against accepted answers. For example, you know the answer to “What was our revenue in 2025?” But does your agent? That’s where your evals come in.

As Anthropic explains in their post demystifying evals for AI agents, there are two types of evals you can run:

• Capability: These evals measure whether your agent can do the task you’re giving it well. For example: can your agent provide accurate revenue numbers? Can it quote the correct average revenue per store? Having a good overview of these questions will show you your agent’s weak spots.

• Regression: You need to make sure that every step forward doesn’t come with two steps back. That’s where your regression evals help out. For example, after you add more context or change your documentation, does the agent still get the revenue numbers right? What about after that naming change or that new model? Is the agent still passing the evals, or have you spotted any backsliding?

Of course, you can’t create an eval for every potential question your stakeholders might ask your data agent. That would be impossible. However, this doesn’t mean you should just give up and accept that your agent is an impenetrable black box.

Think about it this way. You’re suddenly thrown into a vast, dark forest without a map (or cell phone coverage). You have two options: YOLO it and run around, or try to find your way out systematically. That’s what evals give you: an admittedly imperfect but important way to measure your progress toward your goal.

Now, let’s swap the forest for a hilly landscape, put on our hiking boots and start our climb.

What is hillclimbing all about?

There are very few instances where names in the industry make sense. “Hillclimbing” is one of those rare exceptions. The name already provides us with a clear mental picture. Just imagine. You are at the bottom of the mountain (or hill) and your goal is to get to the top.

Now, let’s translate this picture into our agentic world. No matter which tasks we introduce agents to, most likely they will perform well in some areas and really poorly in others. Knowing where these areas are gives us our first picture: our baseline. It will help us identify what we need to modify. From there, the goal is to go uphill.

Things are no different in analytics. Introducing agents as interfaces with which our stakeholders interact implies making sure we know what our agents can do well and where they need assistance. Additionally, we need to develop a proper way to test whether our interventions are actually helping. Evals give us both.

The process involves four steps: setting up a baseline, identifying where to improve, making one group of changes at a time and evaluating them, and iterating for further improvements.

Set up your baseline

To set your baseline, you need three things:

• First, come up with a relevant set of questions from your stakeholders. As data specialists, you’ve already got this. You know which questions are constantly coming to you that your agent has to get right. To look for possible blind spots, check your warehouse and dashboards; they will help you discover the main queries your stakeholders are running. If you have already set up your data agent, ask your stakeholders. Can they point you to cases where the agent got something wrong?1

• Second, you need to decide which method you are going to use to grade your agent’s answers. This is done on an answer-by-answer basis. In some cases, you’d want to see whether the agent is giving you the right number or text. In others, you’d also want to evaluate the quality of the explanation.

• Third, provide the right answers. This is what your agent will be tested against.

The third point is less straightforward than you’d think. While building my own evaluation set, I noticed how open-ended some of my questions were and how much knowledge is tacit. For example, a question that might seem simple at face value, like “What was our revenue last month?”, has more than one interpretation. To start with, what is revenue? But also, how do we calculate last month? Is it the last 30 days, the last 28 days or the last calendar month?

The solution is not to make your questions more precise. On the contrary. Remember, good evals should get as close as possible to the way your final users are interacting with your agent. And there are very strong reasons to believe your stakeholders are not asking: “Please agent, provide me with a sum of our income before taxes for the last 28 days.” That’s what evals are all about: you want to test your agent exactly as your final users would and not spell things out for it. Instead, decide what your business means by “revenue” and “last month,” and write that into your expected answer. Teaching your agent those definitions is part of the climb. Also, make sure the environment you are running your evals in matches the one from your final user as closely as possible.

Identify where you need to improve

Once you have your eval set (aka golden set), go ahead and run it. You should run your evals more than once.2 Why? Remember that agentic systems are non-deterministic. In each run they can yield a different answer. Also, if you are using an LLM judge as the grading method, randomness will affect its grading as well.

After you have run your evals a number of times, you will observe that some groups begin to form based on their outcomes:

• Always passing: your eval passes in every run. These evals won’t help you identify places where your agent can improve; however, they should be part of your regression evals. They can help you identify if your agent is backsliding.

• Never passing: your agent fails its eval on every run. This is your main case for improvement.

• Flip flops: your eval passes in some runs and fails in others. Either your data agent or your grading model is showing inconsistency. Identify the source of the inconsistency and check if remediation is possible.

There’s no better way to understand what your agent is doing than by digging into its traces. Your agent's traces record all the steps it took and the tools it called when trying to provide an answer. I would normally start with the never-passing and flip-flop evals. Digging into the traces will help you answer: Is the agent using the right tables? Is it correctly interpreting the units in the tables? Do multiple sources in your warehouse confuse your agent about which one it should use? In other words, the traces will help you spot what needs to be changed to increase your agent’s passing rate.

At this point, it is very tempting to jump straight into action and start fixing what your agent did wrong. Resist the urge. Before that, check your passing evals. If, in the process, you discover that some passing marks are not well-deserved or your evals are to blame for some of the failures, you need to do some calibration.

Calibration is the process of adapting your evals and/or expected answers to make sure they are really covering what you want them to.

For example, it might be that your eval is passing for the wrong reasons. Say you asked your agent to give you the top three labels in your dataset, and you use a simple text match (contains) as your grading method. If the names of those three labels come up in the answer but aren’t named as the top three, your test will silently pass. Also, you might discover that your agent’s answer is correct, but the grading is failing it. For example, you have identified that your agent should only use certain sources from your warehouse. However, it turns out it can also get to the same answer by using others. You should decide whether you want to allow your agent to use that route. Additionally, your expected answer might be too vague, which leads to inconsistent grading.

The idea behind calibration is that sometimes failures have nothing to do with the agent itself, but with the way you have set up your evals. To repeat a point I made before: it does not mean shaping the eval and framing the question in a way no stakeholder would ask. It also does not mean changing your expected answer to match what the agent said; otherwise your agent will be speaking in an echo chamber. The expected answer should be adjusted to what the business means.

Once you’ve calibrated and re-run your evals, congrats! You now have a baseline. We are at the bottom of the hill right now, with an admittedly incomplete but useful map of how to climb it.

Make one group of changes and evaluate them

Your baseline will give you some paths for improvement. Here’s a piece of advice. Don’t pursue all the paths at the same time. You’ll probably get lost. You want to check whether your intervention is having the desired effect. If you change five different things at the same time, could you say which one did the trick?

For that reason, think about identifying and collecting the changes you want to apply in logically consistent groups. For example, if your baseline was determined by the evals you ran after connecting your agent to the raw data layer, a logically consistent next step could be: what happens if we document the raw layer? As a further step, you can figure out what extra benefit you’ll get by cleaning up your data and generating some models. You can then move on to adding more documentation, a semantic layer, etc. I’d start simple, first with the steps that demand the least effort. By making small incremental changes, you might realize in the process that the more expensive steps are not really worth it.

After adding your changes, it’s eval time again. Run the same evals you had when creating your baseline. Run them more than once to deal with their intrinsic variance.3 Compare the evals one by one and check what their status is across the runs. Are the ones that were failing passing now? How about the ones that were constantly flipping? Did you find any regressions?

How do you identify real improvements? Check your passing evals and compare them to the baseline. Note, again, that there’s a randomness component to evals, so make sure you don’t mistake noise for signal (e.g. all your improvements could just be flip-flop evals that happened to pass in every run). You have a strong case for improvement when a previously never-passing eval passes in all runs. If your flip flops are now consistently passing, check the traces to understand whether this new stability is well-founded.

Look beyond the number of passing evals. The truth might be hidden behind it. If you pull back the curtain, you might discover that your previously failing evals are now passing, but some that were always passing are now flip-flopping or failing. Even if, in the best case, you see an overall increase in passing evals, you might have inadvertently introduced some regressions. To catch them, check your always-passing evals first: they’re your regression evals.

Armed with this evidence, you have to decide whether you keep the change, and this becomes your new baseline, or whether you want to undo it because it is leading you in the wrong direction.

That’s what hillclimbing is all about. It is a series of small conscious steps, where you apply changes, you run your evals and then you check whether there’s an actual improvement. And you keep on repeating this same process, again and again, in your quest to get to the top.

Are we there yet?

It’s natural to think that, at a certain point, after all the steps we have taken in the right direction, we should reach the top. Finally, it’s time to sit down and enjoy the view.

Well, not really. Data agents work in constantly changing environments, and the evals need to keep up with that. Here are some reasons why the process of running evals and hillclimbing should be a constant part of your development flow:

• Remember that your evals only cover a very small subset of the infinite number of questions your stakeholders can ask. If you keep improving on the same set of evals, you will end up overfitting: your agent will get perfect scores on your eval set, but that might not carry over to the actual questions your users are asking.4 To avoid this, rely on your users to inform you about cases where your agent is falling short of the task. Additionally, if possible, constantly inspect some real interactions of users with your agent to find failures. This will allow you to create new evals and will be key for further accuracy gains.

• Your data is always changing. New tables or data models are added, old models are retired, new corner cases need to be documented. Each of those is a new reason to check your evals and see whether there are new areas that you need to cover.

• AI is always changing. New, more capable models are constantly coming out. Don’t assume that new models behave like old ones. Use your evals to prove it.

• By this point, you should already have a number of evals that your agent should always pass. These evals allow you to identify regressions. Constantly run your evals so you can quickly react in case you observe some backsliding.

“The top” is always moving, as you discover new ways in which you need to improve your agent. And that’s good. Every step you take comes with a lesson about your agent, and earns it a bit more of your stakeholders’ trust. Miley Cyrus said it best: “There’s always gonna be another mountain / I’m always gonna wanna make it move / Always gonna be an uphill battle… It’s the climb.”

Let’s do some hill climbing together!

In the coming articles, we will climb together. We are going to put the theory developed here into practice. We will start with a data agent connected only to the raw layer and see how much we improve from our baseline when we start adding documentation, models and better instructions.

We will use The Honest Agent, an open-source tool I created that helps with running and reviewing evals for your data agent. You can find the repo here. Install it and follow along!

If you already work with dbt, The Honest Agent will feel very familiar (e.g. simple commands like run and --select). And, just to keep things extra dbt, we are going to use its best-known dataset: the Jaffle Shop.

And now, let’s start climbing!

“Let’s connect our warehouse to Claude.” I welcomed the suggestion with both excitement and trepidation. At that time, I remember being equally in awe of the opportunities it would open as alarmed by everything that could potentially go wrong. Let me give you a free tour of what went through my mind.

The part of me that got excited saw everything this would make possible.

In most organizations, the data team is seen as the one standing between other teams and the insights they need. “Bottleneck” and “gatekeeping” are not an unusual pairing when someone is talking about the data team. “Isn’t that dashboard or metric simple enough? Why do I need to create a ticket that will be thrown into the backlog, anyway?” At last, the data would be at the stakeholders’ fingertips.

I was also excited for the data team. Gone would be the days of doing the menial work of writing yet another ad-hoc query, creating really complex dashboards no one would look at, or pulling out fresh numbers so they could be imported into an Excel sheet.

A real win-win.

On the other hand, I also saw how opening up the gates could be problematic. In my experience, the best measure of success of a data team is whether they can build trust in the metrics they produce. A well-governed data architecture. Music to my ears.

Trust in data is built by being consistent and predictable. Metrics need to be deterministic. You don’t want last quarter’s revenue to jump from one day to the next. You don’t want 20 different dashboards reporting 20 different numbers for the same KPI.

AI agents are, by nature, probabilistic. Give your agent instructions to write a paragraph. The same instructions can generate multiple versions. Which one is the right one? For text, it might not matter, but for data, it does.

The main challenge in the age of agentic analytics is to give your agent the right harness. You need to constrain its behavior and make it as deterministic as possible.

Yet I felt there was so much out there that focused on the cosmetic work and the presentation layer (did someone say dashboards?) while overlooking what’s inside.

That’s what motivated me to create The Honest Agent. I certainly want to reap the benefits of agentic analytics. But I recognize that this is only possible if people can trust the answers the agent gives them.

Do you actually know whether your agent is being honest, or are you blindly trusting the numbers it gives you?

How does the Honest Agent work?

Remember the last time you had to take a test? You were given a set of questions that would be graded against the accepted answers.

This is exactly what The Honest Agent does. As simple as that. You first define the set of questions, ideally business-critical ones (the ones you really don’t want the agent to get wrong), and then provide the accepted answers for them. Then it’s quiz and report time.

If you already use dbt, The Honest Agent will feel very familiar. Think of it as dbt for your agent evals: you go from running your evals to the report in three simple commands.

I will briefly go through the main components here. Check the docs if you want to dig deeper.

Connect to your MCP server

The Honest Agent connects to your analytics MCP server, the same one your users reach through tools like Claude Cowork.

I’ve tested the setup with Snowflake and MotherDuck. It should work with any MCP server you can reach with a URL and a token. If yours connects another way, reach out and I’ll look at adding it.

Choose your testing model

The Honest Agent needs an LLM to talk to your analytics MCP server and to grade the answers. For this, you need an API key from Anthropic or OpenAI, depending on which provider your organization uses. Note that this is an API key, not a Claude or ChatGPT subscription.

When selecting a model, check which ones are used in your organization (e.g. Claude Sonnet 5.5 or Opus 5.5). This gets you closer to the answers people are getting. The Honest Agent also lets you compare results across models (more on that later).

Note that each evaluation run incurs costs, and these vary with the model you select.

Define your evals

Defining your evals is just a matter of writing them down in YAML. Each eval can check two things:

• Accuracy: is the agent giving you the right answer?

• Provenance: is the agent looking at the right sources?1

Why check both? They point to different issues.

• Your agent might ace accuracy and fail provenance. For example, instead of the table with the aggregated and curated numbers, it looks at some intermediate ones. It got to the right answer, but do you want your agent relying on those intermediate tables?

• Your agent might use the right sources and still fall short on accuracy. It consulted the right tables as part of its work, but got lost along the way.

In my experience, good evals are the ones that cover the questions your users are constantly asking. Take some time to define them. Dig into your warehouse logs and talk to your stakeholders to find out. You won’t be able to cover all the questions, but you can try to cover the most important ones.

Check the docs for details on how to write evals and all the options around them.

Run your evals

You can run your evals with one simple command:

honest-agent run

The run command also offers options to run only a subset of your evals, in the same style as dbt, e.g. honest-agent run --select tag:smoke. Check the docs to learn more.

Build and serve your report

Call me biased, but the report is what all the evaluations are about. A score can only tell you so much if you can’t dig deeper into the questions that failed, the SQL that was run and the steps your agent took.

Building and serving the report is just two commands away:

honest-agent report
honest-agent serve

The report has three sections:

Overview

In one screen, you can quickly compare your agent’s scores. The heatmap gives you an overview of the different eval runs and helps you to spot drifts; click any cell to open that answer. The Below threshold list shows the evals that didn’t pass, and why they failed.

Model comparison

Want to check how different models compare on your evals? Say you want to see how Sonnet 5.5 scores against Haiku 5.5. Or you want to check how OpenAI models compare to Anthropic’s. This is your page.

The heatmap lets you spot the performance gaps across models. You can also track each model’s scores over time. As your data and setup change, you’ll see which model keeps up, and whether a cheaper one is good enough for the questions your team actually asks.

Result page

Admittedly, this is the page where I spend most of my time. It’s like opening a window into your agent’s decision-making process. You can get there by clicking an eval’s cell in the heatmap, or from the Below threshold list in the Overview.

Besides a closer look at how the agent scored on accuracy and provenance, it helps you understand the causes behind those scores.

This is also where The Honest Agent differs most from general-purpose eval tools, which mostly judge the answer as text. In analytics, the answer is a number that came from a query, so The Honest Agent looks at the query too:

• Every SQL query the agent ran is extracted and shown to you, so you can quickly check its logic.

• That SQL is parsed to find the tables it actually read, and compared with the sources your team trusts. Getting the right number from the wrong table shows up as a failure, not a pass.

Make it part of your CI/CD pipelines

The evals are only useful if you keep running them. Because The Honest Agent is a CLI, it fits easily into any scheduler you already use to run your data pipelines. If an eval falls below its threshold, the run fails, so you’ll notice.

You can also configure a Slack webhook to get alerted when your evals fall below their thresholds.

Is The Honest Agent your ultimate solution to build trust in your agentic analytics?

As the creator of a tool that has “honest” in its name, I should hold myself to the same standard: No. The Honest Agent won’t be your ultimate solution, but it can certainly help you along the way.

There are two limitations you should be aware of:

• Each eval is a single, fresh question. In the real world, users interact with agents in a series of questions and answers, where context builds up and shapes the next reply. Multi-turn evaluation is something I want to explore in coming iterations.

• You can only catch errors in the questions you are tracking. Think about the millions of questions your users could ask of your data. It is impossible to foresee and evaluate all of them.

Even with these limitations, The Honest Agent already sheds some light on what used to be a black box. Think about it: if your agent fails to answer a simple question with a fresh context, it is most likely also getting it wrong in longer conversations with your users. And while you can’t catch every error your agent might make, wouldn’t it be great to catch the most important and most frequently asked ones? This is where The Honest Agent can help.

Try it out!

The Honest Agent is open source. You can check the repo on GitHub, and follow the setup guide here. If you’d like to see it in action first, have a look at the demo report.

Note that it’s still in alpha, so there might be some breaking changes while I keep developing it. I’d also be very happy to get your feedback: feel free to reach out in the comments or on LinkedIn.

During those first rides you would get used to the feeling of being on a bike but more importantly they would be full of safety instructions: how to cross the road, how to signal that you are turning and how to be aware of other vehicles. Till the day arrived, when the training wheels came off and without even knowing, off you went.

Harness engineering is like teaching your agent how to ride a bike. It’s those knee pads, helmet, training wheels and all those rules you needed to learn way back then to make it possible to ride a bike without harming yourself. Analogously, an agent is not only the model (aka bike) but also the controls, information, and guidelines it has to follow to arrive at a good destination.

In mathematical terms:

In bike terms:

The importance of the harness: going beyond models

Harness is the whole ecosystem your model is packed with: e.g. tools, context management, basic rules. A model without a harness is, essentially, a clever chatbot.

Time after time, it has been proven that with a better harness, open models can operate and even surpass the performance of frontier models. Take for example the findings from Faros AI. Across 211 real engineering tasks spanning 12 repositories, they tested several model-and-harness combinations, and the open-weight models didn’t just match the frontier ones, they actually outperformed them on quality, at roughly half the cost.

This finding holds just as true in the agentic data world. By this point, it is a known fact that, no matter how advanced your model is, if you feed it messy and ambiguous data, it will confidently set you up for failure. Less complex models beat more powerful ones when they have access to clear, structured, and documented data, which is really just another way of saying a better harness.

Anthropic’s work on their self service analytics is testament to that. According to their own account, accuracy went from below 21% to consistently above 95%, and what drove that jump was work on the harness: decreasing data ambiguity, improving modeling, creating a semantic layer, and building structured skills. Using another agent to aggressively challenge the results, meanwhile, added only 6% accuracy at the cost of 32% more tokens and 72% higher latency.

What goes into building an agentic analytics harness?

Another way of thinking about the models that companies like Anthropic or OpenAI make available is to consider them as general purpose models. Going from a general purpose model to a highly specialized one in your specific area is a matter of providing it with the right harness.

When talking about harness engineering one might get the false impression that it’s a single discipline. Surely, there are some principles that apply across the board, like tracing, observability, and validation, but the way you build your harness is highly domain dependent. After all, the whole point is specializing your agent for a particular job. An agent that is great at debugging your backend application might fail at the simplest data retrieval task.

In analytics, the goal is clear: you want your agent to get to the right answer. The problem: there’s only one such right answer. The harness you build needs to constrain the non-deterministic traits inherent in the agent’s behavior. Given a question, the agent needs to point to the correct answer, over and over. There’s a lot of work to get there, as attested in Anthropic’s own post and this Thoughtworks piece on getting your data ready for agentic AI.

So far, my impression is that the debate has been focused very much on the side of context and information retrieval, and for very good reasons. Context plays a very important role in analytics. A well-maintained context reduces ambiguity and keeps your agent honest.

However, concentrating only on context risks ignoring other important elements of the ecosystem that agents operate in and that constrain their behavior. That’s why I prefer speaking about harnesses. Sure, context is part of the harness, but there are also other components, all posing their own set of challenges for steering your agent’s actions:

Agentic Analytics demands data harness engineering

How is AI changing a data practitioner’s role? As more and more analytics work is delegated to agents, we will soon start seeing where this approach falls short. An agent will always provide you with an answer. But can you trust it?

Building those specialized harnesses will be key to using data agents reliably. Future data practitioners need to understand how to work with agents and how to build and evaluate the harnesses these agents are working with.

This means looking far beyond modeling and documentation and having an understanding of how agents work: How do agents generate their context? How can we generate better skills? Do we have the tools to check what answers they are providing to our stakeholders? Do we have appropriate tests to check for quality drifts?

In other words, to borrow from the metaphor this article started with, data practitioners are those parents teaching the agent how to ride the bike. Our work is now getting the agent properly equipped to make the ride successful.

You think back, smiling, to that bygone era when you needed to contact the overwhelmed data analyst, create a ticket, and pull every lever to get it prioritized. Data is now where it needs to be: at the heart of the company, pumping every decision, one Slack message away.

You share the news with finance. Instead of celebration, you’re met with a chill. Those new subscriptions you were so excited about never turned into actual revenue. Right. It was too good to be true. A dream, or technically speaking, a hallucination.

Agentic Analytics: the quest for following the right path

Think about the time you asked your favorite AI tool to build a document or presentation for you. You might need to do some small tweaks, change the tone, and so on. But there were many different ways the task could have been done well.

Data is different. Your task is to land on a single answer (say, the number of subscriptions) and that answer has to be correct.

As Anthropic puts it in their article on enabling self-service analytics with Claude, some domains, like coding, have a much wider solution space (the range of answers that would all be acceptable). In data, the path is narrow. Coding is an open-ended space where creativity pays off, and tests and docs catch wrong answers for you. Analytics is the opposite: there’s often, in their words, “only a single correct answer using a single correct source,” with no deterministic way to prove it’s right.

The main task in this new agentic analytics work is to narrow the path the agent takes and lead it to the right solution. The good news is that most of this can already be built with your current stack, if you’ve been following good data practices. Being ready for the agentic era means doubling down on modeling, documentation, and validation. As Anthropic found, that’s the difference between an agent that’s right 21% of the time and one that’s right 95%.

3 pillars for Agentic Analytics

In the new agentic world, context is king. Context, in my view, is the ecosystem in which the agent operates and makes its decisions. In the data world, two pillars build that context: data modeling and documentation. A third one keeps it honest: validation, making sure the context is correct and stays that way as the business changes.

Data modeling foundations

We’ve all been there. You need to put together a quick report on subscriptions. You go digging and find five different datasets that could work, each giving a slightly different number. So you muster some courage and go straight to the source, trying to connect the tables yourself. You spot a c_id (customer, maybe?) but nothing obvious to join it to. You give up. You go for one and hope for the best.

Agents are no different. Five models all covering the same concept don’t narrow the agent’s path to the right answer, they widen it. And when it isn’t clear how those models connect, the way that c_id had nowhere obvious to join, the agent is left guessing the same way you were.

The bottom line: creating data models for agents is no different from creating them for humans. Think of them as data products. They should be intuitively easy to work with, where dimensional modeling earns its keep. They should reduce ambiguity, so no two models do the same thing. Your organization’s metrics should be codified in a semantic layer, so they resolve to a single definition. And tests and freshness checks keep your sources reliable.

Documentation

Remember that surge in subscriptions that never turned into revenue? Finally your analyst had some time to check it out. The data was “technically” correct, but the surge came from a migration of old accounts to the new subscription provider.

If you’ve worked with data, you know the numbers you see are only part of the truth. The business context and the processes that produce them matter just as much. This kind of knowledge is cumulative, scattered across the organization, and often not written down anywhere.

This is where documentation sets a different bar in the agentic world. It’s not just keeping your dataset descriptions up to date. It means capturing all that process and business logic, the things that usually live in people’s heads, in a written form an agent can actually retrieve and use. Think about the documentation you wish you’d had when you were onboarded.

Your documentation should cover at least three things. First, your data sources: what each one is, what its columns mean, how it relates to others, and what events to account for when interpreting results (remember the subscription migration?). Second, business context: the campaign that drove a spike in registrations, the roadmap change that shifted a metric, the things that explain why the numbers look the way they do. Third, skills.

Skills, as Anthropic describes them, are the procedural knowledge that tells the agent how to navigate everything you’ve documented: which sources to consult first, how to handle ambiguity, and when to hold back rather than answer from a missing or conflicting source. It’s the layer Anthropic credits with the largest gains in accuracy.

Validation

Let’s say you’ve invested the time and effort: proper documentation, business context, sharpened agent skills. Is that it?

That’s only part of the work. Data sources and agent development move fast, and your context goes stale with them. Anthropic watched their own accuracy drift from 95% to 65% in a single month before they started actively maintaining it. So you need to keep your context current and keep checking that your agent’s quality hasn’t degraded over time.

A best practice here is to move context and modeling into the same repository and treat context like any other code change. You can also add a skill that checks, whenever you update a data model, whether the documentation needs updating too.

Another important part is constantly checking your agent’s accuracy. Think of it like quizzing it on the same questions you already know the answers to. Then, when you ship a new model version or edit a skill, you can see whether it actually helped or quietly made things worse.

Can we guarantee the agent will always get it right? Not really. At least, not yet. If you think about it, there’s a catch in that 95% accuracy Anthropic reports, one they acknowledge themselves. You can only measure accuracy on questions you already know the answer to. The silent failures, by definition, are the ones that never make it into your test set. You can’t quiz the agent on a mistake you don’t know it’s making. But that’s why validation is central, and never finished. You can’t eliminate the silent failures, but you can keep shrinking the space where they hide: catching drift, harvesting corrections, sanity-checking the numbers that matter most.

Conclusion

The agentic analytics era is here to stay. As I’ve argued, whether agents succeed in your organization hinges on reducing ambiguity and narrowing their options down to the single right answer.

That demands doubling down on some good old data practices, like building intuitive data models and investing the time to document them. It also means adopting new ones: curating your business context, sharpening your agent skills, and keeping everything honest with continuous validation.

And it demands accepting a trade-off. Democratizing data through agents means there will be cases where someone trusts a wrong answer without ever realizing it. The real challenge ahead isn’t building the agent, it’s making sure that when it’s wrong, you find out before a bad decision gets made.

That world, where your 7am Slack digest steers you right instead of wrong, is a dream well within reach. But it isn’t built by pointing an agent at your warehouse and hoping for the best. What separates dream from hallucination? It’s the unglamorous work underneath: the models, the documentation, the validation. Get it right, and it stays a dream.

At this point, a small confession. I welcomed all the excitement around quack, however I was struggling to understand what impact it would have on my own work. I went back and forth on my current workflow trying to identify use cases. I did find some small optimizations, but nothing that would match all that excitement in my timeline.

And, then, in one of those episodes where the answer had been in front of you the whole time but you failed to realize it, I had my aha moment: our dbt workflow. Quack is the missing piece DuckDB needed to make dbt team workflows work.

Let me explain. So far, DuckDB was great at what it promised. It was a super optimized analytics warehouse that you can run on your machine. Meanwhile, dbt is now the de facto tool most data teams use to transform their raw data in meaningful models. Your DuckDB + dbt combo workflow normally involves reading data from some cloud storage provider (like s3) and performing all the transformations on your local machine.

While this worked very well when developing in isolation, things got complicated quite quickly the moment you wanted to work in a more collaborative setup. Let me clarify this with an example.

Let’s say Ana is currently working on a new customer churn model. dbt and DuckDB means that she can work in a very isolated way and run all the models on her machine. That’s great. The problem starts when, let’s say her colleague Alex, wants to inspect Ana’s model, to validate her calculations. The complete development isolation had one considerable tradeoff: no simple way to share Ana’s work. Alex would have to ask for her file.

This is exactly what quack changes. It allows DuckDB instances to talk to each other. For dbt, this means that now Ana can run her models in a common DuckDB database shared by the team. If Alex needs to check Ana’s work, he can simply, for example, check dev.ana.customer_churn. In other words, quack has opened the possibility for dbt development flows that are more collaborative and akin to the experience developers have when working with other warehouses, like snowflake or BigQuery with possibly one big twist.

Unlike traditional warehouses, with dbt + DuckDB + quack developers can use their local machines to do the transformations and then write those changes to the shared DuckDB instance when ready. This also has a meaningful cost implication: computation happens on the developer’s own machine, not on a cloud warehouse billing by computation time. Compared to running every dbt model against Snowflake or BigQuery, the savings can be significant.

If you are a data team running on a small budget, this could be the stack you were looking for. Spin up a small shared DuckDB instance, configure your dbt targets, and you are good to go. Additionally, developers can take advantage of their local machines to build their models, something a traditional cloud warehouse simply doesn’t allow. Just keep in mind, quack is still in its initial stages and under development. However, the promise is there: your dbt development will soon get quacking and cheaper.

You know the feeling. Just when you are wrapping up the meeting about the definitely last changes to a report — the last ones, they swear — you are distracted by a Slack notification in the management channel: “@here, don’t forget that next Tuesday we are going to discuss our next quarter’s OKRs. Start adding your suggestions! 😊”

At this point, you have already started scratching your head, trying to come up with some possibilities. You already know the theory. Yes, the objective needs to be inspiring. Yes, you want your key results to be SMART! By then, you have probably forgotten what each letter of that acronym means, except for the M (hint: measurable). After all, aren’t metrics exactly what the data department is good at?

After writing a couple of sentences, you soon discover it is a dead end. A couple of hits to the backspace key puts you back at the beginning. Clean slate. After trying again, you hit the backspace key again. This time, harder. Yes, setting OKRs as an analytics team is frustrating. There’s something in the whole OKR setup that does not quite seem to capture the wide array of activities an analytics team regularly conducts.

OKRs: the case for radical focus

The real power of OKRs stems from their capacity to focus efforts around one or two bold objectives a company wants to achieve within a defined timeframe. That is Christina Wodtke’s argument in her book Radical Focus. It is this sense of clarity and alignment that allows companies to track progress, stay accountable, learn and celebrate all the actions contributing towards their goals.

If you are a data lead reading Wodtke’s book, you probably won’t find that any purely analytics objective should make it to the winners’ podium of your company’s OKRs. And that’s fine. Don’t get me wrong, analytics and reporting tasks are really important, but they are not the core of your business. In Wodtke’s terms, these are supporting teams, much like finance or legal.

Making OKRs work for your analytics team

So, how can you tie your analytics team back to the OKRs and reap the benefits they are set to bring? After all, there are two important things OKRs provide: focus and a sense of accomplishment. These are key drivers for team productivity and morale.

There are a couple of ways.

Working with the company OKRs, making them your own

Emphasize both within and beyond your team how your work is supporting the company’s OKRs.

Let’s say this quarter’s objective is all about conversion. Why not highlight that dashboard your team created to help track conversions? Or the tracking setup your team put together? Or the experiment you helped to run?

These are all meaningful contributions your team made that helped drive this OKR. In this way your team will not only internalize that OKR (and as a by-product help them prioritize their work) but also help them establish a direct connection between their work and the company’s larger objectives; something analysts consistently struggle with.

Creating your own team OKR, tuning your processes

Think about that team process that is not quite working yet. For example, you have identified serious gaps in the data onboarding process you provide for other teams that you want to close. Or, perhaps, your experimental setup is not quite where you want it to be.

These are all activities worth turning into a proper objective. If you decide to go that route, there are a couple of things to keep in mind.

First, make sure you get buy-in from leadership. After all, OKRs are about sending a strong message about what you want to focus on and should help you prioritize. It will also keep you and your team accountable. A good way to secure that buy-in is to make a clear connection between the process gap you want to address and the company’s quarterly OKRs.

Second, remember that this is a team OKR and your team is there to support the company. If you ever have to choose between your team’s OKR and the company’s, the choice is simple: company OKRs always come first.

These two strategies are not mutually exclusive. You can do both. The first one (working with the company OKRs) should be part of your regular quarterly exercise. The second one (tuning your processes) is not a must. You can turn to it whenever you feel your team and organization could benefit from some process tuning.

Radical support to achieve radical focus

OKRs can be wonderful when done correctly. In Wodtke’s framework, they enable your organization to radically focus on those one or two objectives you want to achieve. For analytics teams, this means admitting that your work is there to support the company and probably won’t be directly reflected in a company goal.

This is not a bad thing. It does not mean your team’s contributions are unimportant, or that OKRs have nothing to offer your team. The trick is doubling down on your supportive role. As great films need great supporting actors, companies need radical support to achieve radical focus. And there is nothing small about that.

Part of our role as managers is to foster a harmonious working environment that lets teams thrive. It was not easy realizing that the purported equilibrium I tried to maintain was not there in the first place. After licking my battle wounds, I realized: actually this person is giving me the chance to act before the worst outcome: “I quit”.

This realization brought me back to one of my favorite political science books: Exit, Voice, and Loyalty by Albert Hirschman. In his work, Hirschman explores different strategies people deploy in the face of a deteriorating product. As the title says, we can either remain using the product despite its lower quality (loyalty), buy another one (exit), or raise this issue to the supplier (voice).

Hirschman offers a compelling critique of why exit, normally emphasized as a market correcting mechanism, is not the only way people show their discontent. By voicing discontent, people are giving a chance for course correction.

The parallels in teams and organizations are striking. In the face of deteriorating conditions, members can decide to stay silent, quit and find another job, or raise their voice to enact change.

This is no different in any team, and as a manager you probably want to avoid exit. Here there are two mechanisms at play, one is within your realm of control and the other one is not. Let’s start with the latter.

People’s ability to exit depends on the exit options. In other words, if the job market is bad (as it currently is) people do not actually have a high chance of exiting. In the long term, you cannot rely on a bad market to keep your organizations together because conditions change quickly. Retention built on the scarcity in the market is just a delayed exit.

The second mechanism is where you have a real lever. You can encourage voice by lowering its cost. Companies try to do this by institutionalizing processes, like 1:1s, surveys, and feedback loops. But these are only as good as the trust underneath them. What actually lowers the cost of voice is the belief that speaking up won’t be held against you, and that someone is genuinely listening.

Voice is a real gift, and it works on two levels. First, it is a testament to the trust the person is placing in you. Second, it provides you with invaluable information on what is not working and what you can do to correct it. The next time someone on your team says ‘I’m not happy with the direction we are going,’ consider yourself lucky.

Determined to crack open what big data was all about, I enrolled in the big data coursera track. Through a series of videos and online tutorials, engineers very excitedly went into great detail about big data formats, computational architecture, clusters, nodes and many other words that were foreign to my novice ears. The peak of the course came when we were tasked to write our own MapReduce job in Cloudera. For me it felt like a lot of hard work just to get a simple count.

Then luck struck. Twice. First, I did get a data analyst role. And second, the job did not entail writing MapReduce jobs! However, the big data movement was very much still alive touting its next promise: warehouses. The promise was enticing: forget about managing clusters. Simply concentrate on what you can do best: writing elegant SQL.

That promise has held up for the most part. I have spent most of my career working with different warehouses; starting with Redshift, followed by a brief stint with BigQuery and currently working with Snowflake. Despite some minor differences, the common thread is that I can simply concentrate on writing SQL while the underlying architectural complexity is hidden away from me.

Warehouses work well. However, this simplicity comes with a price tag. Sometimes, a big one. A colleague of mine learned it the hard way. A single query to get a monthly count was racking up hundreds of dollars. To limit this potentially unchecked spending, companies have to set up a whole system of guardrails, query optimization and resource monitoring.

So while warehouses made it technically easy to store and access your data, the real barrier was costs. Big data comes with a big budget. And big budgets are nowhere to be found in startups and mid-size companies.

Is there another way? In his post big data is dead, Jordan Tigani, co-founder of MotherDuck and former founding engineer of Google BigQuery, certainly believes so. He makes two compelling arguments:

• You probably don’t have big data

• Even in the unlikely case you have big data, most of your queries are accessing only a small fraction of it

That begs the question: when is data big? A simple answer is whenever you cannot process or analyze it on your own laptop. Turns out that according to a Redshift fleet study analyzed by Tigani, only around 5% of datasets in Redshift actually have big data. The remaining 95% can probably be analyzed on your own machine.

Meanwhile, our laptops have quietly become more powerful and their computational capacity has grown at a faster speed than our data. Data for which originally the MapReduce framework was invented can now often be analyzed on your laptop in seconds.

Even when our data is big, it is generally the case we are not accessing all of it at once. Think about how data is used in your company. With the exception of some deep dive analysis, the most accessed data is usually the most recent. This is the data that powers your dashboards and normally is already pre-aggregated in some kind of daily, monthly or yearly metrics.

This was also found in the Redshift analysis cited before. Only 0.03% of queries executed by customers were against more than 10TB of data. A whopping 67% were querying less than 1GB worth of data.

If we don’t really have big data, has the industry been optimizing for the wrong use case all along? In other words, are we building our whole infrastructure around a Ferrari when probably a reliable Toyota would do the job?

This is exactly the bet DuckDB and MotherDuck are making. Their promise relies on squeezing the most out of your single machine. Once you remove all the unnecessary architectural complexity, queries are not only faster but also substantially cheaper. If you do not have big data — and most likely you don’t — it is time to reconsider. Are you sure you need the costly Ferrari, or has the time come to switch to a Toyota?

This thought hit me as I was going through different YouTube videos about strategy, OKRs and all the big words that are now an established part of startup parlance. I had a moment of déjà vu. Wait a minute. I have heard this before.

I was suddenly thrown back into my first research methods classes and this simple yet poignant question my advisor would drill me with: so, José, what is the question you are trying to answer?

Turns out I was not alone back then. In the industry there are committees, boards and countless meetings trying to nail down what problem their product is actually solving.

And, again, to quote my advisor: the methods should follow your question and hypothesis and not the other way around. This is no different in the industry. The mission, vision, strategy, OKRs and roadmap should follow your question and hypothesis and not be a replacement for the lack of one.

As planning for the new quarter starts and you find yourself buried in sticky notes, Miro boards and workshops, trying to translate it all into strategy documents, OKRs and initiatives, connect with your inner advisor and ask yourself: what is the problem we are trying to solve here? Just as in school, answering might be difficult and thought provoking, but in exchange everything that follows will come naturally from it.