No. This is not an article about hiking. And, sorry to disappoint Miley Cyrus fans, but it’s not about her either. It’s actually about both, and about neither. It’s about running evaluations (evals) for agentic analytics and why using methods like hillclimbing can be key to improving your data agent’s accuracy.
But first, let’s start with a dramatization, based on real life, of a company that has just connected Claude to their warehouse (reader discretion is advised).
Clara: “I just updated the investor deck. It was super easy this time! I asked Claude to get the numbers directly from the data. And I’ve been creating these cool dashboards too. Look!”
Three hours later…
Clara: “Just double-checked with the data team. Turns out the numbers were completely wrong. Good thing I hadn’t sent the deck yet.”
Clara was lucky this time, but who knows how many people out there are blindly trusting whatever their agent confidently presents to them as facts. Without a way to verify your agent’s output, you’re in for a wild ride through uncharted territory. Evals help you regain control and visibility over your agent’s actions.
The idea behind evals is simple. There’s no better way to understand how your agent performs than giving it a series of tasks and grading its output against accepted answers. For example, you know the answer to “What was our revenue in 2025?” But does your agent? That’s where your evals come in.
As Anthropic explains in their post demystifying evals for AI agents, there are two types of evals you can run:
Capability: These evals measure whether your agent can do the task you’re giving it well. For example: can your agent provide accurate revenue numbers? Can it quote the correct average revenue per store? Having a good overview of these questions will show you your agent’s weak spots.
Regression: You need to make sure that every step forward doesn’t come with two steps back. That’s where your regression evals help out. For example, after you add more context or change your documentation, does the agent still get the revenue numbers right? What about after that naming change or that new model? Is the agent still passing the evals, or have you spotted any backsliding?
Of course, you can’t create an eval for every potential question your stakeholders might ask your data agent. That would be impossible. However, this doesn’t mean you should just give up and accept that your agent is an impenetrable black box.
Think about it this way. You’re suddenly thrown into a vast, dark forest without a map (or cell phone coverage). You have two options: YOLO it and run around, or try to find your way out systematically. That’s what evals give you: an admittedly imperfect but important way to measure your progress toward your goal.
Now, let’s swap the forest for a hilly landscape, put on our hiking boots and start our climb.
What is hillclimbing all about?
There are very few instances where names in the industry make sense. “Hillclimbing” is one of those rare exceptions. The name already provides us with a clear mental picture. Just imagine. You are at the bottom of the mountain (or hill) and your goal is to get to the top.
Now, let’s translate this picture into our agentic world. No matter which tasks we introduce agents to, most likely they will perform well in some areas and really poorly in others. Knowing where these areas are gives us our first picture: our baseline. It will help us identify what we need to modify. From there, the goal is to go uphill.
Things are no different in analytics. Introducing agents as interfaces with which our stakeholders interact implies making sure we know what our agents can do well and where they need assistance. Additionally, we need to develop a proper way to test whether our interventions are actually helping. Evals give us both.
The process involves four steps: setting up a baseline, identifying where to improve, making one group of changes at a time and evaluating them, and iterating for further improvements.
Set up your baseline
To set your baseline, you need three things:
First, come up with a relevant set of questions from your stakeholders. As data specialists, you’ve already got this. You know which questions are constantly coming to you that your agent has to get right. To look for possible blind spots, check your warehouse and dashboards; they will help you discover the main queries your stakeholders are running. If you have already set up your data agent, ask your stakeholders. Can they point you to cases where the agent got something wrong?1
Second, you need to decide which method you are going to use to grade your agent’s answers. This is done on an answer-by-answer basis. In some cases, you’d want to see whether the agent is giving you the right number or text. In others, you’d also want to evaluate the quality of the explanation.
Third, provide the right answers. This is what your agent will be tested against.
The third point is less straightforward than you’d think. While building my own evaluation set, I noticed how open-ended some of my questions were and how much knowledge is tacit. For example, a question that might seem simple at face value, like “What was our revenue last month?”, has more than one interpretation. To start with, what is revenue? But also, how do we calculate last month? Is it the last 30 days, the last 28 days or the last calendar month?
The solution is not to make your questions more precise. On the contrary. Remember, good evals should get as close as possible to the way your final users are interacting with your agent. And there are very strong reasons to believe your stakeholders are not asking: “Please agent, provide me with a sum of our income before taxes for the last 28 days.” That’s what evals are all about: you want to test your agent exactly as your final users would and not spell things out for it. Instead, decide what your business means by “revenue” and “last month,” and write that into your expected answer. Teaching your agent those definitions is part of the climb. Also, make sure the environment you are running your evals in matches the one from your final user as closely as possible.
Identify where you need to improve
Once you have your eval set (aka golden set), go ahead and run it. You should run your evals more than once.2 Why? Remember that agentic systems are non-deterministic. In each run they can yield a different answer. Also, if you are using an LLM judge as the grading method, randomness will affect its grading as well.
After you have run your evals a number of times, you will observe that some groups begin to form based on their outcomes:
Always passing: your eval passes in every run. These evals won’t help you identify places where your agent can improve; however, they should be part of your regression evals. They can help you identify if your agent is backsliding.
Never passing: your agent fails its eval on every run. This is your main case for improvement.
Flip flops: your eval passes in some runs and fails in others. Either your data agent or your grading model is showing inconsistency. Identify the source of the inconsistency and check if remediation is possible.
There’s no better way to understand what your agent is doing than by digging into its traces. Your agent's traces record all the steps it took and the tools it called when trying to provide an answer. I would normally start with the never-passing and flip-flop evals. Digging into the traces will help you answer: Is the agent using the right tables? Is it correctly interpreting the units in the tables? Do multiple sources in your warehouse confuse your agent about which one it should use? In other words, the traces will help you spot what needs to be changed to increase your agent’s passing rate.
At this point, it is very tempting to jump straight into action and start fixing what your agent did wrong. Resist the urge. Before that, check your passing evals. If, in the process, you discover that some passing marks are not well-deserved or your evals are to blame for some of the failures, you need to do some calibration.
Calibration is the process of adapting your evals and/or expected answers to make sure they are really covering what you want them to.
For example, it might be that your eval is passing for the wrong reasons. Say you asked your agent to give you the top three labels in your dataset, and you use a simple text match (contains) as your grading method. If the names of those three labels come up in the answer but aren’t named as the top three, your test will silently pass. Also, you might discover that your agent’s answer is correct, but the grading is failing it. For example, you have identified that your agent should only use certain sources from your warehouse. However, it turns out it can also get to the same answer by using others. You should decide whether you want to allow your agent to use that route. Additionally, your expected answer might be too vague, which leads to inconsistent grading.
The idea behind calibration is that sometimes failures have nothing to do with the agent itself, but with the way you have set up your evals. To repeat a point I made before: it does not mean shaping the eval and framing the question in a way no stakeholder would ask. It also does not mean changing your expected answer to match what the agent said; otherwise your agent will be speaking in an echo chamber. The expected answer should be adjusted to what the business means.
Once you’ve calibrated and re-run your evals, congrats! You now have a baseline. We are at the bottom of the hill right now, with an admittedly incomplete but useful map of how to climb it.
Make one group of changes and evaluate them
Your baseline will give you some paths for improvement. Here’s a piece of advice. Don’t pursue all the paths at the same time. You’ll probably get lost. You want to check whether your intervention is having the desired effect. If you change five different things at the same time, could you say which one did the trick?
For that reason, think about identifying and collecting the changes you want to apply in logically consistent groups. For example, if your baseline was determined by the evals you ran after connecting your agent to the raw data layer, a logically consistent next step could be: what happens if we document the raw layer? As a further step, you can figure out what extra benefit you’ll get by cleaning up your data and generating some models. You can then move on to adding more documentation, a semantic layer, etc. I’d start simple, first with the steps that demand the least effort. By making small incremental changes, you might realize in the process that the more expensive steps are not really worth it.
After adding your changes, it’s eval time again. Run the same evals you had when creating your baseline. Run them more than once to deal with their intrinsic variance.3 Compare the evals one by one and check what their status is across the runs. Are the ones that were failing passing now? How about the ones that were constantly flipping? Did you find any regressions?
How do you identify real improvements? Check your passing evals and compare them to the baseline. Note, again, that there’s a randomness component to evals, so make sure you don’t mistake noise for signal (e.g. all your improvements could just be flip-flop evals that happened to pass in every run). You have a strong case for improvement when a previously never-passing eval passes in all runs. If your flip flops are now consistently passing, check the traces to understand whether this new stability is well-founded.
Look beyond the number of passing evals. The truth might be hidden behind it. If you pull back the curtain, you might discover that your previously failing evals are now passing, but some that were always passing are now flip-flopping or failing. Even if, in the best case, you see an overall increase in passing evals, you might have inadvertently introduced some regressions. To catch them, check your always-passing evals first: they’re your regression evals.
Armed with this evidence, you have to decide whether you keep the change, and this becomes your new baseline, or whether you want to undo it because it is leading you in the wrong direction.
That’s what hillclimbing is all about. It is a series of small conscious steps, where you apply changes, you run your evals and then you check whether there’s an actual improvement. And you keep on repeating this same process, again and again, in your quest to get to the top.
Are we there yet?
It’s natural to think that, at a certain point, after all the steps we have taken in the right direction, we should reach the top. Finally, it’s time to sit down and enjoy the view.
Well, not really. Data agents work in constantly changing environments, and the evals need to keep up with that. Here are some reasons why the process of running evals and hillclimbing should be a constant part of your development flow:
Remember that your evals only cover a very small subset of the infinite number of questions your stakeholders can ask. If you keep improving on the same set of evals, you will end up overfitting: your agent will get perfect scores on your eval set, but that might not carry over to the actual questions your users are asking.4 To avoid this, rely on your users to inform you about cases where your agent is falling short of the task. Additionally, if possible, constantly inspect some real interactions of users with your agent to find failures. This will allow you to create new evals and will be key for further accuracy gains.
Your data is always changing. New tables or data models are added, old models are retired, new corner cases need to be documented. Each of those is a new reason to check your evals and see whether there are new areas that you need to cover.
AI is always changing. New, more capable models are constantly coming out. Don’t assume that new models behave like old ones. Use your evals to prove it.
By this point, you should already have a number of evals that your agent should always pass. These evals allow you to identify regressions. Constantly run your evals so you can quickly react in case you observe some backsliding.
“The top” is always moving, as you discover new ways in which you need to improve your agent. And that’s good. Every step you take comes with a lesson about your agent, and earns it a bit more of your stakeholders’ trust. Miley Cyrus said it best: “There’s always gonna be another mountain / I’m always gonna wanna make it move / Always gonna be an uphill battle… It’s the climb.”
Let’s do some hill climbing together!
In the coming articles, we will climb together. We are going to put the theory developed here into practice. We will start with a data agent connected only to the raw layer and see how much we improve from our baseline when we start adding documentation, models and better instructions.
We will use The Honest Agent, an open-source tool I created that helps with running and reviewing evals for your data agent. You can find the repo here. Install it and follow along!
If you already work with dbt, The Honest Agent will feel very familiar (e.g. simple commands like run and --select). And, just to keep things extra dbt, we are going to use its best-known dataset: the Jaffle Shop.
And now, let’s start climbing!
Ideally, you have some observability of your agent in production. This will help you catch the errors your stakeholders might not even be aware of.
The number of times you want to run your evals is decided case by case. I recommend you do it at least three times (as I will do in the example I will present). Three runs give you a rough indication of which of the three groups your eval belongs to (always passing, never passing and flip flops). Note that running evals incurs token costs. There’s a trade-off to be made here between minimizing the variance of your passing rate and the budget you are willing to spend on it.
See previous footnote.
In the following articles we will use a technique known as holdout to reduce the risk of overfitting.