Your Better Model Is Worse at Your Tools: The Harness Is Load-Bearing

Armin Ronacher upgraded to the newest frontier models, the ones topping every benchmark, and watched his coding agent get worse. Not slower. Worse at the one thing an agent has to nail: calling tools correctly. The new models started inventing fields that did not exist in his schema (requireUnique, oldText2), the kind of malformed call the older models did not make. His writeup, amplified by Simon Willison the same day, hit the front page of Hacker News because a lot of people had quietly noticed the same thing and assumed it was just them.

It was not just them. And the upgrade was not a fluke. It is the predictable result of how these models are built, and it points at a layer of engineering most teams have been treating as a vendor detail.

Why does a newer AI model make more tool-calling errors than an older one?

Because a model's tool-use skill is not a property of the model alone. It is a property of the model plus the harness it was trained against. A frontier model gets heavily reinforcement-trained against one specific coding harness, usually the vendor's own edit tools, which are forgiving and quietly repair slightly-malformed calls. The model learns that one schema deeply and treats yours as off-distribution.

So it emits calls that would be valid against the harness it grew up in, and invalid against yours. The fluency is real. It is just fluency in someone else's interface. Some of what a new generation buys with all that training is depth in the vendor's surface, and depth in one surface is bought partly at the expense of generality. The newer model is not dumber. It is more specialized, in a direction that is not yours.

This is a cousin of a problem we have written about before: model behavior you cannot fully inspect. You do not get to see the training distribution. You find out where it lands by running your own tools against it and watching what breaks.

What is a coding-agent harness?

The harness is everything around the model: the tool definitions and schemas, the edit format, the error messages, the retry logic. It is the surface the model acts through. It feels like plumbing, the boring layer beneath the intelligence. It is not boring. It is frequently the dominant variable in whether an agent works at all.

Here is the number that should end the "just upgrade the model" reflex. The practitioner can.ac ran a controlled eval: hold the fleet of models fixed, change only the edit-tool schema, and measure. Swapping the schema moved Grok Code Fast 1 from 6.7% to 68.3% on the same coding tasks. Same model. Same problems. Roughly a 10x swing, produced by the harness alone. Across 16 models and three edit formats, the pattern held: the weakest models gained the most from a better schema, which is the tell that the harness had been masking capability all along.

Sit with that title. Fifteen models got dramatically better at coding and not one of them was retrained. The intelligence did not move. The surface did. If a single schema choice can be the difference between a 6.7% agent and a 68.3% agent on the same model, then the harness is not a detail you abstract away. It is the load-bearing wall.

Which reframes Ronacher's bad week. Part of what the newest models spend their "capability gain" on is fluency in a proprietary harness, not general tool use. The frontier is getting better at the vendor's own surface and, at the margin, a little worse at yours.

Is it lock-in if a model works best inside its vendor's own coding tool?

It is a lock-in gradient, not a hard lock. Nobody is trapped. A model trained against its vendor's harness simply performs best there and charges a reliability tax everywhere else. You can pay that tax down, but the fact that you have to engineer around one vendor's conventions to get your own tools working is the lock-in showing up as a line item.

And here is where the strongest counter-argument comes in, the one all over the Hacker News thread: this is fixable at the harness layer. Return a helpful error and the agent self-corrects on the retry. Name your tool Edit, not edit, so the casing matches what the model expects. Validate the malformed call and auto-correct it instead of rejecting it. The critics are right. It is fixable.

That concession does not weaken the argument. It is the argument. Every one of those fixes is harness engineering. "Just fix your harness" quietly concedes that the reliability you get is the reliability you designed into the action surface the agent operates through. The model did not abstract that layer away for you. It handed it back.

How do you make an AI coding agent more reliable at tool calls?

Treat the harness as engineering, not a footnote. Four moves do most of the work. Return corrective error messages so the model repairs itself on the next turn. Match the tool names and casing the model was trained on. Validate and auto-correct near-miss calls rather than failing them. And measure tool-schema adherence across model versions, so you catch a regression before your users do.

None of this is exotic. It is the boring layer, schemas and error contracts and retries and naming, which is exactly where reliability has always lived. This is the lesson the AI era keeps re-teaching in new costume: the fundamentals did not get deleted, they got concentrated. Agentic coding did not remove the difference between a 6.7% and a 68.3% agent. It moved that difference into the harness and made it enormous.

The deeper consequence is about how you evaluate a model at all. If capability lives in the model-plus-harness pair, you cannot judge a model in the abstract. A leaderboard score against the vendor's own tools tells you almost nothing about how it will behave against yours.

Does upgrading to the latest AI model always improve results?

Not for tool use against a custom harness. Newer models can regress on non-native schemas even while they improve on reasoning and general benchmarks. It is not that newer is worse. It is that newer is not automatically better for the specific tools you run, and the only way to know is to check.

So before you upgrade a production agent, run your own tool-adherence eval on the new version against your actual schema. Treat "the new model is better" as a hypothesis to test on your harness, not a release note to trust. This lands, not coincidentally, during a week when the same frontier convenience is showing its bill on the pricing side too: the same convenience that has a schema tax also has a variable-cost tax. Two seams, technical and economic, and builders own both.

The model will keep getting better at someone's tools. Whether it gets better at yours is an engineering decision, one you make deliberately or one a vendor's training objective makes for you by default. The harness is load-bearing. Build it like it is.