What an agent harness is made of

A coding agent is an artificial intelligence (AI) model plus the program that runs it. That program, the harness, decides what the model sees, what it may do, how its work is checked and what it remembers. Change the harness and the same model can solve a very different share of the same tasks, so it pays to know what a harness is made of and how to compare two of them.

First principles

A model predicts the next tokens from the ones it is given. It cannot open a file, run a test or remember yesterday; the harness does all of that. Five facts about the model set the harness’s job.

1. It acts only through text. Its output has to be parsed into an action, and the result turned back into text. So a harness needs a loop and a tool surface.
2. Its window is finite, and it gets worse as it fills. So a harness needs context management.
3. It cannot reliably tell when it is wrong. So a harness needs checks from outside the model.
4. It can mistake what it reads for instructions, and acts with whatever authority it has. That includes text planted in a web page or a file. So a harness needs permissions and a sandbox.
5. It forgets everything between calls. So a harness needs state.

That gives six parts, each of which can be built well or badly whatever the model.

The loop

The basic loop is usually traced to the Reasoning and Acting (ReAct) paper: the model reasons, picks an action, sees the result and repeats. Model Evaluation and Threat Research (METR) uses it as one of its two default scaffolds, and most coding agents share its shape. Three decisions remain.

• Who chooses the next step. In Anthropic’s terms, code fixes the sequence in a workflow and the model chooses in an agent. A workflow is more predictable and easier to test. An agent handles tasks nobody planned for.
• Whether to plan separately. METR’s Triframe has an adviser plan, several actors propose actions, and raters pick the one to run.
• When to stop. The model declares itself done, the harness caps turns, tokens or time, or a check decides. Caps are accuracy settings: an agent cut off at its limit scores the same as one that was wrong. We made the same point about endpoints in The same weights are not the same model.

The tool surface

A tool is a name, a description, a parameter schema and an output format. The model sees all four as text and learns the tool from that text alone. Anthropic describes tools as a contract between deterministic software and a non-deterministic agent.

The clearest measurement is still the SWE-agent paper, which introduced the idea of an agent-computer interface (ACI). Holding the model fixed, the authors changed one part of the interface at a time. An editor that ran a syntax check before accepting an edit beat the same editor without it. A file viewer showing 100 lines beat one showing 30 lines and one showing the whole file. Search that summarised matches beat search that paged through them. Each worse choice cost between a sixth and a third of the tasks the default solved.

Anthropic reported the same in its SWE-bench work: it spent more time on tools than on the prompt, and a model that misused relative paths stopped once the tool required absolute ones. It also advises truncating output by default and writing errors that say what to do next.

Two habits follow.

• Keep output stable and honest. Our search tool, glep, replaces an agent’s Grep and Glob with indexed versions. Its docs make byte-for-byte compatibility with ripgrep’s output the first constraint, because agents and hooks parse that output. The second is that the index must never lie: when it cannot rule a file out, glep searches the file, because an agent acts on a wrong answer as readily as a right one.
• Fail open, and visibly. glep’s Claude Code hook never runs a search itself. When glep is installed and the working directory has an index, it blocks the built-in tool and names the equivalent command, so the agent sees exactly what ran. Otherwise, or when glep cannot honour the request exactly, it lets the built-in call through.

Context management

The window holds the system prompt, tool definitions, instruction files, the conversation and every tool result. It is a budget. Chroma tested 18 models and found performance changed with input length even on simple tasks. And every turn resends the whole window, so cost grows with history.

Between them, Anthropic’s guide to context engineering and Manus’s account of its agent describe five techniques.

1. Load just in time. Keep file paths and read content through tools when needed.
2. Clear old tool results. SWE-agent kept only the last five observations in full and scored better than with the whole history.
3. Compact. When the window fills, summarise the session, keep decisions and open bugs, and continue in a fresh window.
4. Delegate. A sub-agent explores in its own window and returns a short summary.
5. Keep the prefix stable. Providers cache the start of a prompt, and cached input is much cheaper. A timestamp at the top, or tools added and removed mid-task, breaks the cache. Manus masks unavailable tools instead of removing them.

Memory that outlasts one window overlaps with state. Our Trove sits at the simple end: one small file per decision or convention, an index with one line per entry that a session-start hook loads, and code context retrieved from the top down within a token budget. Its matching is lexical, so it misses synonyms, as its docs say.

Verification and feedback

A model asked to recheck its own reasoning, with no new information, does not reliably improve it. A 2023 study of self-correction found answers sometimes got worse after revision. Anthropic’s advice is to get ground truth from the environment at each step.

Checks form a ladder, from cheap and strict to costly and loose.

• Deterministic checks. The code builds, the linter and the tests pass, the migration reverses.
• End-to-end checks. In Anthropic’s long-running agent experiments, agents marked features done after unit tests, and browser automation that used each feature found bugs the code did not show.
• Critics. On chat answers, the strongest model judge in one study agreed with people about as often as people agree with each other, but judges tended to favour the first answer shown and longer answers, and may favour their own outputs.

This failure usually shows only in the logs. When the check is a test suite, an agent can satisfy the test instead of the task. The Holistic Agent Leaderboard (HAL) team found agents that hard-coded plausible answers to pass unit tests and agents that looked up benchmark answers online.

Our Kestrel, a coding command-line interface (CLI) whose first version is still in build, follows this ladder. Its site describes harness packs that end in a deterministic checklist, and an --ultra mode where attempts run in separate Git worktrees, three judges read every diff, a fourth tallies votes and the winner repairs the caveats. A panel lowers the odds that one bad read decides, but not biases the judges share, so the deterministic checks come first.

Permissions and sandboxing

Anything the model reads can steer what it does. Simon Willison calls the dangerous combination the lethal trifecta: private data, untrusted content, and a way to send data out. An agent with all three can be tricked into leaking.

Harnesses answer with approval and sandboxing. Approval asks a person before an action, but asking too often causes what Anthropic calls approval fatigue. A sandbox limits what any action can reach. Anthropic’s sandbox for Claude Code restricts both the filesystem and the network with operating system primitives, and argues both are needed: without network limits a compromised agent can send out keys, and without filesystem limits it can escape to reach the network.

A sandbox makes it safe to ask less. Kestrel’s site describes the common split: read-only tools never prompt, anything that writes asks first, and a rule the user chooses to always allow is saved to a settings file. One limit applies to any harness built on Git worktrees, ours included: a worktree separates the checkout, not the filesystem, network or credentials.

Permissions also change scores: block the network and tasks that need it fail; allow it and an agent can look answers up.

State

Within a task, state is the message history. Across sessions, it is what the harness writes down. Anthropic’s long-running agent harness had a first session write a start-up script, a progress log and a list of more than 200 features, all marked failing. Each later session made one increment, committed it and updated the log. It targeted two failures: an agent that builds everything at once and runs out of context, and a later agent that sees partial progress and declares the job done.

The environment is state too. Benchmarks reset it between tasks; real repositories do not.

Why the same model scores differently

Anthropic put it plainly: SWE-bench “doesn’t just evaluate the AI model in isolation”; it measures the whole agent system. The six parts explain the gaps.

• Interfaces fail silently. A misread output or a mishandled submission turns correct work into a zero. HAL’s generalist agent scored close to zero on one coding subset with two OpenAI models that solved nearly half of it under SWE-agent. HAL does not give the cause, but a gap that large points at the harness, not the model.
• Harnesses fit model families. HAL reported that Claude models did better with one browser scaffold and OpenAI models with another. A pre-registered study on the General AI Assistants (GAIA) benchmark found model family, not capability tier, predicted which scaffold helped.
• Graders differ. On CORE-Bench Hard, Claude Opus 4.5 nearly doubled its score when moved from the task-specific CORE-Agent scaffold to Claude Code, and fixing errors in the automatic grading by hand raised it further. The grader is part of the measurement.
• Budgets differ. Turn, token and time caps decide how many hard tasks finish.
• Noise. On a 50-task subset, one task is two points. An audit of the SWE-bench leaderboard could not separate any adjacent pair in the Verified top thirty.

The strongest counterargument is that once a harness is sound, the choice matters much less. A note published by METR compared Claude Opus 4.5 under Claude Code with ReAct, and GPT-5 under Codex with Triframe, at equal token budgets, and found neither difference significant. The authors of mini-SWE-agent report strong SWE-bench results from an agent class of about 100 lines of Python with only a shell for tools. And AI Agents That Matter found that a simple retry baseline matched the best elaborate agent on HumanEval and cost less than most of them.

We think both are right about different things. The large gaps come from broken interfaces, mismatched pairings, graders and budgets, and elaborate orchestration on a sound harness adds less. The GAIA study is the main exception: on its harder level, structured scaffolds helped the most capable Anthropic model the most, and its author concludes that nothing guarantees the gap shrinks as models improve.

How to compare harnesses fairly

1. Change one thing. Fix the model, endpoint, decoding settings and version. Swap only the harness.
2. Keep tasks, environment and grader identical.
3. Match budgets, or plot accuracy against cost and compare the frontiers.
4. Find the noise floor. Run the reference twice, and use paired tests, since both harnesses see the same tasks.
5. Read the logs for hard-coded answers, leaked answers and silent failures before trusting a score.
6. Test each model in a neutral harness and in its best one. One pairing describes the pair, not the model.
7. Hold tasks out. Do not tune on the tasks you report.
8. Report provenance: model, version, harness, budgets and date.

What to measure

We decide with cost per accepted result; this paper explains why.

What to do

• Start with a plain loop and a few well-described tools. Add structure only when logs show a failure it fixes.
• Treat tools as interfaces, and rerun the same tasks after every change.
• Cap tool output, clear old results and keep the prompt prefix stable.
• End every task with a deterministic check. Put critics on top, not instead.
• Sandbox filesystem and network together, then cut approvals.
• Before switching models, try the same model in another harness.

As of 2026-10-25

HAL has paused adding models and notes that some reported costs are overstated. Most HAL figures above are single runs without intervals.

What we could not verify

• Why HAL’s generalist agent scored 0% and 2% with o3 and GPT-4.1. HAL does not say, and we did not read the logs.
• How the CORE-Bench gain splits between harness and grading. HAL does not separate them or give an interval.
• Terminal-Bench 2.0. Secondary sites report swings of 20 points or more across harnesses. The official leaderboard did not render for us.
• Codex internals. OpenAI’s post on the Codex agent loop returned an access error.
• Kestrel. We describe what its site says. It has no published measurements against other harnesses yet.

Sources

Harness design: Building effective agents, Raising the bar on SWE-bench Verified, Writing effective tools for agents, Effective context engineering, Effective harnesses for long-running agents, Claude Code sandboxing, Manus on context engineering, mini-SWE-agent, The lethal trifecta.

Papers: ReAct, SWE-agent, AI Agents That Matter, Holistic Agent Leaderboard, Scaffold effects on GAIA, Coding agents have converged, Models cannot self-correct reasoning yet, Model judges on MT-Bench and Chatbot Arena, Context rot.

Leaderboards and notes: HAL CORE-Bench Hard, HAL SWE-bench Verified Mini, METR on Claude Code and Codex.

Velofy: glep, glep with Claude Code, How glep works, Trove, Trove memory hierarchy, Kestrel.

← All journal entries