Agent harnesses: what they are, and how to check them

Sairah Jahangir

Agent = model + harness

A large language model on its own (GPT, Claude, Gemini and so on) can only answer the message it is given. To turn it into an agent that can carry out a task over many steps, you have to wrap it in software. That wrapping is now called a harness.

A harness typically includes:

  • The system prompt: standing instructions the model reads before every conversation, telling it who it is and how to behave.
  • Tools: the actions the agent is allowed to take, such as searching the web, reading a file, sending an email or querying a database. Each time the agent uses one, that is a tool call.
  • Memory: what the agent is allowed to remember between steps or between sessions.
  • Hooks: small pieces of code that run automatically at set moments, for example a check that fires just before the agent uses a tool and can block the action.
  • Permissions: rules about what the agent may and may not touch.

Birgitta Böckeler of the software consultancy Thoughtworks puts it simply: agent = model + harness. Well-known examples are Anthropic's Claude Code, OpenAI's Codex CLI and LangChain's DeepAgents. ("CLI" means command-line interface: the tool runs in a text window on a developer's computer rather than in a web browser.)

OpenAI popularised the term in February 2026 with an article on "harness engineering", and the idea has spread fast. By September, the software firm Marmelab had counted 21,500 new harness projects created this year on GitHub, the main website where developers publish code. In one week in late August, four of the ten fastest-growing open-source AI projects on GitHub were harnesses or frameworks for constraining agents. The leader, DeepSeek Harness, had almost 188,000 "stars", GitHub's equivalent of likes or bookmarks.

What a harness does, and doesn't do

The engineer Philipp Schmid compares a harness to a computer's operating system. The model is the processor. The context window, the amount of text the model can hold in view at once, is the working memory. The harness decides what goes into that memory and which tools are available. The agent is the application that runs on top.

Building a harness is a construction job. Its author chooses the tools, writes the rules and sets up the hooks. Whether the finished agent then behaves well with a real customer is a separate question, which cannot be wholly determined by the harness. Schmid makes a similar point about benchmark scores: a model that tops a league table by one percentage point is no use if it loses track of the task after fifty steps.

Most harnesses are released untested

Marmelab's report, The State of AI Harness Engineering 2026, examined 97 harnesses large enough to assess. 60% had neither automated tests (checks that the code works as written) nor evaluations, usually shortened to evals (checks of how well the agent actually performs its task). Only seven had both.

Safety controls look even thinner. Of 391 projects examined, just 12 included a single deny rule, a hard setting that blocks the agent from a specific action whatever it decides. A separate study looked at 481 public CLAUDE.md files. CLAUDE.md is a plain-text file of instructions that Claude Code reads at the start of each session, such as "never delete files in this folder" or "never share customer data". Only 4.4% of the security instructions in those files were backed by an actual technical control. The rest were sentences in a prompt that the agent was simply trusted to read and obey.

There is a further problem, which practitioners call harness rot. A harness is built around the weaknesses of the model it was designed for. When the model is upgraded, some of those workarounds become pointless, and others start to get in the way. Anthropic found that logic it had built to periodically clear and refresh the model's working memory became unnecessary once its Opus 4.5 model arrived. The agent company Manus rebuilt its harness five times in six months. Every rebuild changes how the agent behaves, and most teams have no set of tests to re-run afterwards to see what changed.

VerifyAX is the genuine test

Conscium's VerifyAX tests the finished agent, regardless of which harness was used to build it.

You connect your agent to VerifyAX using one of two standard methods: A2A (Agent-to-Agent, an open standard that lets AI agents talk to each other) or a REST API (the common way web services exchange data). You then choose test scenarios from the VerifyAX library, searchable by the skill being tested, or write your own. VerifyAX runs simulated conversations with your agent. Characters controlled by the platform play the parts of customers, colleagues and attackers. A separate AI model, acting as a judge, reads the full record of each conversation and scores where the agent fell short.

It is independent of the harness. A harness that checks its own agent is marking its own homework. VerifyAX brings its own scenarios and its own hidden model answers, including questions designed to tempt an agent into making things up.

It catches harness rot. You can save a set of scenarios and run exactly the same set after every model upgrade, prompt change or new tool. Because the tests are identical each time, the scores are directly comparable, and you can see which skills got better, which got worse and which stayed the same.

It turns written rules into evidence. VerifyAX's policy compliance tests take the rules you have written for your agent and check whether the agent actually follows them. A line in a CLAUDE.md file stops being a hand-wavy wish, and becomes a firm criterion in a test, with the conversation record to back up the results.

It checks whether the agent behaves differently when it thinks it is being tested. Some agents act more safely when a prompt is obviously a test than when it looks like a real request. VerifyAX runs both versions of the same scenario and flags any gap between the scores.

It works inside the tools developers already use. The VerifyAX plugin for Claude Code connects a Claude Code agent to VerifyAX and runs scenarios against it without leaving the developer's working window. For other harnesses, VerifyAX offers an MCP server. MCP (Model Context Protocol) is an open standard for plugging outside services into AI tools, and it lets VerifyAX work inside Codex CLI and Google's Gemini CLI. Developers can test in the same session in which they build.

Five questions to ask before trusting an agent built with a harness

  1. Who tested the agent? Was it the same team that built the harness?
  2. Which scenarios were tested? Did they include hostile users and questions designed to trip the agent up?
  3. Which of the rules in the agent's instruction file have been checked against what the agent actually does?
  4. When the model or the harness last changed, were the same tests run again, and did the scores hold up?
  5. Is there a written record of the testing that your risk or compliance team will accept?

Conclusion

Harnesses determine how quickly a team can build and release agents. They do not show that an agent is fit for release. That evidence has to come from testing that the agent did not design for itself. Run your next agent through VerifyAX before it goes live, and run the same tests again after every change.

Download our whitepaper to explore 50 common AI agent failure modes and how to test for them.

Download the Whitepaper

Sources

  • The State of AI Harness Engineering 2026, Marmelab, 24 September 2026
  • The importance of Agent Harness in 2026, Philipp Schmid, 5 January 2026
  • AI Open-Source Radar, 17–24 August 2026, AgentLensHQ