AI-Native Methodology

CLAUDE.md Files Make Coding Agents Cheaper, Not More Correct

Bill Cava/

Every team building with coding agents writes the file. AGENTS.md now sits in over 60,000 open-source projects and has become a cross-tool standard.[3] Andrej Karpathy's CLAUDE.md guidelines became one of the most-shared templates in the ecosystem. bun, a popular JavaScript toolchain, now creates a CLAUDE.md for you when it sets up a new project.

And when an agent disappoints, the community's first prescription is always the same: improve your context file.

Someone finally measured whether that works. The answer splits cleanly in two, and neither half is what the ritual assumes.

Do context files make coding agents more correct?

Not measurably. A July 2026 study ran two leading coding agents through 288 evaluated attempts at 17 real tasks across 3 repositories, varying only how the project's instructions were supplied, and found no measurable difference in how often the agents got the change right.[1]

The study, by researcher Prakhar Khatri, is a controlled comparison of one variable at a time: same agents (Claude Code and Codex), same tasks, same repositories, scored against each repo's own known-good tests. The only thing that changed between runs was the context file. Full instructions, no instructions, and variations in between.

The sharpest detail is what happened with the repositories' real instruction files, the ones their maintainers actually wrote. Those files never converted a near-miss into a pass. Not once, on either agent.

The findings text of the July 2026 arXiv paper on context files and coding agents
The paper's own summary of the result: context strategy does not measurably move correctness, and the failure analysis explains why. Source: arXiv:2607.27250.

Be precise about what this does and does not show. The bound is wide: it rules out big effects (roughly 10 percentage points or more), not small ones. And it is one paper, not yet peer-reviewed, from a single author, on three repositories and one generation of agents, with all code and data released for anyone to check.

So the defensible claim is "no measurable correctness gain," never "context files are useless." We hold measured claims to a higher bar than vibes, and this one stays inside its evidence.

The study also explains why earlier, smaller tests kept contradicting each other. Which tasks sat on the edge of an agent's ability turned out to be specific to each agent (a strong correlation, for the statistically minded: 0.75). Two studies testing two different agents were sampling two different sets of hard problems, and each saw a different picture.

What does an AGENTS.md file actually buy?

Efficiency, and a meaningful amount of it. A separate study from January 2026 measured agents working through 124 real code changes across 10 repositories, with and without an AGENTS.md file present. With the file: median runtime fell 28.64%, and output tokens (the units agent work is billed in) fell 16.58%, with comparable task completion either way.[2]

What the file changed in measured runs, across two 2026 studies: both costs fell, and correctness did not measurably move.

Put the two results together and the picture snaps into focus. The context file is an economics lever, not a skill substitute. It changes what the agent spends, not what the agent can do.

That is a genuinely useful thing. A file you write once that makes every future session faster and cheaper is the best-priced improvement in agentic development.

It is also a different thing than what the ecosystem believes it bought. The harness around a model shapes an enormous amount of how work feels and what it costs. What it does not do, on this evidence, is make the work more likely to be right.

Why do coding agents fail even with a good context file?

Because the failures are mostly skill failures, not knowledge failures. Khatri's failure analysis looked at where the agents actually broke down, and the answer was not "it didn't know where things live." It was design: choosing the right approach, selecting the right pattern, wiring the change correctly.

Agents fail on implementation skill, feature design, pattern selection, exact wiring, not missing repository knowledge that a context file could supply.

Prakhar Khatri, arXiv:2607.27250, July 2026

Read that against what teams put in these files: build commands, conventions, architecture notes. All of it is knowledge. None of it is judgment. The file can tell an agent where the tests are. It cannot tell the agent what a good solution looks like.

We have made this argument from the manifesto side for a year: AI amplifies your direction, right or wrong, and the aim has to come from somewhere. What is new is a measurement drawing the same line. The judgment a Markdown file cannot inject is exactly what the humans in the collaboration bring.

How should you treat your CLAUDE.md now?

Like onboarding documentation for a new collaborator, because that is what the evidence says it is. Write down how to build, how to test, what the conventions are, and the decisions a newcomer could not infer from the code. That is the content the efficiency study rewards.

What the data says to stop doing is iterating on the file expecting correctness to move. If an agent keeps getting a task wrong, another paragraph of context is not the fix. The fix lives in the task, the design, or the judgment applied to it: smaller scope, clearer intent, a human eye on the approach before the agent runs.

There is a pattern forming across the measured levers, and it is worth seeing whole:

Structure and context govern the agent's costs. Correctness still comes from skill. The discourse keeps blurring that line, and the measurements keep drawing it.

The boundary everyone is rediscovering

The week this measurement landed, the front of Hacker News was a run of essays arguing the same point from the human side: don't be a relay between an AI and your codebase, retype what the model writes so you still understand it, and the most important skill in prompting is expertise in the domain you're prompting for.

A July study of five research-focused forums found the same boundary in a different population: students who outsource the cognitive effort "end up with neither the expected results nor the necessary competence."[4]

Different corners of the field, same discovery. Nothing you paste into context substitutes for knowing what good looks like. The file is worth writing, for the same reason good onboarding docs were always worth writing. The judgment was never in the docs. It was never in the code either.

Write the file. Expect the right thing from it.

References

Frequently asked

Do CLAUDE.md and AGENTS.md files actually improve coding-agent results?
›They improve the economics of a coding agent, not its correctness.
⌄They improve the economics of a coding agent, not its correctness. A July 2026 study ran two agents through 288 evaluated attempts on real repositories and found the way project instructions were supplied produced no measurable correctness gain. A separate January 2026 comparison found the file cut median runtime about 29% and output tokens about 17% at comparable task completion. The file makes the agent cheaper and faster, not more likely to get the change right.
How does a CLAUDE.md file work?
›It is a plain-text file at the root of your project that a coding agent reads automatically at the start of a session.
⌄It is a plain-text file at the root of your project that a coding agent reads automatically at the start of a session. Teams use it for build and test commands, code conventions, and architecture notes. The measurements show what that can and cannot change: the file reduces the time and tokens an agent spends rediscovering how your project works, but it does not supply the implementation skill the agent lacks when a task is beyond it.
Why use a CLAUDE.md or AGENTS.md file at all?
›Because the efficiency gain is real, and it repeats on every task the agent runs.
⌄Because the efficiency gain is real, and it repeats on every task the agent runs. In the January 2026 comparison, agents working in repositories with the file present finished in about 29% less time and produced about 17% fewer output tokens, with comparable task completion. One plain-text file, written once, saving time and tokens on every session is the cheapest efficiency lever in agentic coding.
Why do coding agents fail even with a good context file?
›Because most failures are skill failures, not knowledge failures.
⌄Because most failures are skill failures, not knowledge failures. The July 2026 study's failure analysis found agents break down on feature design, pattern selection, and exact wiring: knowing how to make the change, not knowing where things live. The sharpest detail makes the point directly: the repositories' own real instruction files never converted a near-miss into a pass, on either agent.
When should you update your CLAUDE.md?
›Treat it like onboarding documentation, not like a tuning knob.
⌄Treat it like onboarding documentation, not like a tuning knob. Update it when the facts change: new build or test commands, new conventions, architectural decisions worth writing down. What the data says to stop doing is iterating on the file expecting correctness to move. If the agent keeps getting a task wrong, the fix lives in the task, the design, or the judgment applied to it.
What should go in an AGENTS.md file?
›The things a new collaborator would need on day one: how to build the project, how to run the tests, the code style, and the decisions that are not obvious from the code.
⌄The things a new collaborator would need on day one: how to build the project, how to run the tests, the code style, and the decisions that are not obvious from the code. That is also exactly the content the efficiency study associates with faster, cheaper runs. Keep it lean. It is documentation for a collaborator, not a spellbook.
Work with us

Let’s build it together.

We turn clever prototypes into production systems people can rely on. If you’re building with agents and want a hand making it real, leave your email and we’ll be in touch.

Straight to the team. No spam.