Agent Harness Design Matters More Than Your Tool List

On August 12, Microsoft shipped a version of VS Code that pulls agent sessions out of the editor and runs them as their own process.^[2] Twenty-four hours later, DeepSeek open-sourced its own harness under the MIT license, organized around a single claim: everything in it is a plugin.^[3]

The repository passed 100,000 stars within about two days of being created, and has since gone past 185,000.

So the harness stopped being an implementation detail inside somebody's product and became a thing you choose, configure, and replace. That is a real change, and a good one.

It also hands you a decision you did not have a month ago, and most of the commentary stops right where that decision starts.

What is an agent harness?

The harness is everything around the model that lets an agent work in a real environment: how tools are defined and exposed, the edit surface, the session log, the approval policy, the sandbox, and the loop that decides what happens next. DeepSeek puts it as an equation on its own page, Agent = Model + Harness.

The model supplies capability. The harness decides what that capability can reach, and how reliably it lands.

That framing is now shared across vendors, which is what makes this month different. Microsoft is not just shipping a feature, it is using "harness" as a category of interchangeable thing and has published an open specification so that clients can attach to agent backends from more than one company.

What did the study actually change?

On August 11, a team from Purdue, Microsoft Research and the University of Chicago published a controlled experiment that holds model capability constant and varies only how tools are organized and exposed.^[1] Six interfaces, three models, 11,700 runs on real repository issues. Their name for the variable is tool architecture.

The design is the part that matters. Every interface gives the agent broadly the same underlying information and actions, so any difference in what the agent does traces back to organization alone, not to a capability someone added.

In the authors' words, prior work "has primarily focused on expanding what agents can do, but has paid less systematic attention to how those capabilities are organized and exposed to the model."

They also measure the right thing.

The headline metric is pass^k, which the paper defines as the probability that k repeated attempts at the same problem instance are all successful. That is not a resolve rate, and the difference is the whole point: production work depends on an agent doing the same job twice, and benchmarks almost never report it.

Which part of the harness moved the needle?

Organization did. Against a baseline where the agent has only a general-purpose shell tool, more structured low-level interfaces improved consistency across repeated attempts by up to 4.7 times.^[1] That figure is a consistency multiplier, not an accuracy or success-rate claim, and "up to" is doing real work in that sentence.

Here is what sits underneath it, one metric at a time.

Three things are worth reading off that table, and only the first one made the abstract.

The 4.7 times figure is the best case, and it belongs to the weakest model. It is Qwen3Coder-30B going from 0.020 to 0.094 on nine repeated attempts. On the strongest model tested, the same change is worth +0.031. The interface matters most exactly where the model is weakest, which is a useful thing to know before you generalize the number to your own stack.

Splitting one general-purpose tool into narrow purpose-built ones is also the only interface change that improved every model. Plain-language repository search helped two models and hurt one. That is a smaller, less quotable result than 4.7 times, and it is the one you can actually plan around.

The efficiency finding sits on its own axis and should stay there. A Python-style interface, where the agent issues actions as code, reached similar task performance with 41.6% fewer steps and 56.3% lower token usage.^[1]

Worth noting that "similar" is the authors' word for the performance comparison rather than a measured equivalence bound, and that in the consistency table the same interface is flat to negative. Cheaper per attempt, no steadier across attempts.

Does giving an agent a scratchpad help?

Not much, on this evidence. The study tested lightweight text-based cognitive-scaffolding tools, meaning a place for the agent to record its intermediate reasoning, and found they "have limited effect on actor behavior."^[1] In the consistency table, the scratchpad setup came out slightly negative for all three models rather than merely flat.

This is the result that should change what teams do next, because the scratchpad is the cheapest thing to add, the easiest thing to demo, and the first thing most people reach for once every layer is swappable.

Two honest limits before anyone over-reads it.

This is one benchmark subset of 65 instances drawn from 25 repositories, with three models and no frontier tier above Claude Sonnet 4.5, published as a preprint without peer review.

And "limited effect in this setup" is not "useless everywhere." A scratchpad may well earn its place in a workflow this experiment did not run. What the evidence does not support is treating it as the default first improvement.

Agent harness vs model: where is the leverage?

Both matter, and the harness is the part you own. A separate cross-benchmark decomposition of one production agent system attributed most of its reliability advantage over the frontier base model to the architecture around the model rather than to the base model itself, with its verification step contributing just +1.5 points in isolation.^[4] One system, evaluated over two months, so treat it as directional.

This is the same position we have argued twice already, from different directions. In July we wrote that a newer model can be worse at your tools, because capability lives in the model plus the harness.

Also in July we wrote that structure is the token-cost lever, after one engineer's refactor of a 17,155-line file cut the input tokens an agent needed for the same change by 83%.

Different studies, different quantities, same direction. The shape of the surface beats the size of the pile.

There is a real tension with that first post worth naming rather than smoothing over. It argued that tool skill is harness-relative, because a model trained against one vendor's schema goes off-distribution on yours. This study found interface effects that held across three different models. Both are true. Compatibility is model-specific, design quality is less so, and you need to get both right.

What to do with a layer you now control

The practical version is short, and none of it waits on a better model. Three rules, in the order they pay off:

• Choose the interface shape before you choose the tool list.
• Measure whether an agent can do the same job twice, not whether it did it once.
• Require a new tool to demonstrate a behavior change before it earns a slot.

Skip that third one and you get what the study measured: a stack of layers that demo well and move nothing.

None of this is an argument against what DeepSeek and Microsoft shipped. Open-sourcing a harness under a permissive license and publishing a protocol that hosts competitors' agents is the behavior you want from vendors, and it is the reason this decision is yours at all.

But swappability is a precondition, not an answer. Being able to recompose every layer tells you nothing about which layer deserves your attention, and the first controlled look at that question says the answer is not the layer that is easiest to add.