Remember when you first learned to ride a bike? Your first rides would probably be alongside someone older, perhaps your mom or dad. You would wear protection gear, like knee pads and a helmet. And, if you were like me, your bike would have those small training wheels attached to the back wheel to help you keep your balance.
During those first rides you would get used to the feeling of being on a bike but more importantly they would be full of safety instructions: how to cross the road, how to signal that you are turning and how to be aware of other vehicles. Till the day arrived, when the training wheels came off and without even knowing, off you went.
Harness engineering is like teaching your agent how to ride a bike. It’s those knee pads, helmet, training wheels and all those rules you needed to learn way back then to make it possible to ride a bike without harming yourself. Analogously, an agent is not only the model (aka bike) but also the controls, information, and guidelines it has to follow to arrive at a good destination.
In mathematical terms:
In bike terms:
The importance of the harness: going beyond models
Harness is the whole ecosystem your model is packed with: e.g. tools, context management, basic rules. A model without a harness is, essentially, a clever chatbot.
Time after time, it has been proven that with a better harness, open models can operate and even surpass the performance of frontier models. Take for example the findings from Faros AI. Across 211 real engineering tasks spanning 12 repositories, they tested several model-and-harness combinations, and the open-weight models didn’t just match the frontier ones, they actually outperformed them on quality, at roughly half the cost.
This finding holds just as true in the agentic data world. By this point, it is a known fact that, no matter how advanced your model is, if you feed it messy and ambiguous data, it will confidently set you up for failure. Less complex models beat more powerful ones when they have access to clear, structured, and documented data, which is really just another way of saying a better harness.
Anthropic’s work on their self service analytics is testament to that. According to their own account, accuracy went from below 21% to consistently above 95%, and what drove that jump was work on the harness: decreasing data ambiguity, improving modeling, creating a semantic layer, and building structured skills. Using another agent to aggressively challenge the results, meanwhile, added only 6% accuracy at the cost of 32% more tokens and 72% higher latency.
What goes into building an agentic analytics harness?
Another way of thinking about the models that companies like Anthropic or OpenAI make available is to consider them as general purpose models. Going from a general purpose model to a highly specialized one in your specific area is a matter of providing it with the right harness.
When talking about harness engineering one might get the false impression that it’s a single discipline. Surely, there are some principles that apply across the board, like tracing, observability, and validation, but the way you build your harness is highly domain dependent. After all, the whole point is specializing your agent for a particular job. An agent that is great at debugging your backend application might fail at the simplest data retrieval task.
In analytics, the goal is clear: you want your agent to get to the right answer. The problem: there’s only one such right answer. The harness you build needs to constrain the non-deterministic traits inherent in the agent’s behavior. Given a question, the agent needs to point to the correct answer, over and over. There’s a lot of work to get there, as attested in Anthropic’s own post and this Thoughtworks piece on getting your data ready for agentic AI.
So far, my impression is that the debate has been focused very much on the side of context and information retrieval, and for very good reasons. Context plays a very important role in analytics. A well-maintained context reduces ambiguity and keeps your agent honest.
However, concentrating only on context risks ignoring other important elements of the ecosystem that agents operate in and that constrain their behavior. That’s why I prefer speaking about harnesses. Sure, context is part of the harness, but there are also other components, all posing their own set of challenges for steering your agent’s actions:
Agentic Analytics demands data harness engineering
How is AI changing a data practitioner’s role? As more and more analytics work is delegated to agents, we will soon start seeing where this approach falls short. An agent will always provide you with an answer. But can you trust it?
Building those specialized harnesses will be key to using data agents reliably. Future data practitioners need to understand how to work with agents and how to build and evaluate the harnesses these agents are working with.
This means looking far beyond modeling and documentation and having an understanding of how agents work: How do agents generate their context? How can we generate better skills? Do we have the tools to check what answers they are providing to our stakeholders? Do we have appropriate tests to check for quality drifts?
In other words, to borrow from the metaphor this article started with, data practitioners are those parents teaching the agent how to ride the bike. Our work is now getting the agent properly equipped to make the ride successful.