AI Coding Agents Pass the First Change and Fail the Feature

GitHub's annual Octoverse report landed on Wednesday with a headline number: by September 2026, more than 40% of pull requests on GitHub started with an agent's first draft, up from 17% in March^[1].

A pull request is a proposed change to a codebase, submitted for review before it is merged in. Agents started 110 million of them in September alone, and their one quality signal is that they now merge within two weeks a little more often than the rest.

A day earlier, Tencent's Hunyuan team and The Chinese University of Hong Kong published a benchmark that measures what happens after the merge^[2]. Put the two side by side and the most-cited number in AI coding is counting the wrong thing. A merged change is one step of a feature, and the measured failure shows up across steps.

What did GitHub's Octoverse 2026 say about AI coding agents?

GitHub's Octoverse 2026 says more than 40% of pull requests on the platform now start with an AI agent's first draft, up from 17% in March. Agent pull requests touch more files than the rest and now merge within two weeks slightly more often. The report counts activity and merge speed, not what happens afterward.

The merge numbers are the closest the report comes to judging the work.

That table is honest about its scope. The report has no figures on reverts, defects or how the merged work behaves later, and GitHub calls its own agent detection "a floor, since not all agent use may be detected."^[1]

We have written about what agent pull requests skip on the way in, like the documentation and tests that should change alongside the code. This post is about what a merge cannot see on the way out.

One of the first reactions summed up how the numbers landed.

He is right about the impact. The volume is real. The question is what a merged pull request tells you about the software it belongs to.

Does a merged pull request mean the code works?

A merged pull request means one change passed the checks in place when it landed. Real features grow over several changes, each built on the code from the last. Whether the feature works is a question about the final code after every change, which a merge rate cannot answer.

We made the same argument about model pricing: the unit that matters is the finished task, not the token. Pull requests are the token of software delivery. They are easy to count, and they tell you nothing on their own about whether anything got finished.

So the useful question is narrower. When an agent builds the second, third and fourth change on top of its own first one, does the feature still work at the end?

What happens when AI coding agents keep building on their own code?

FeatureArc tested ten coding agents on real features that grew in two to ten stages. Pooled across agents, 62.7% passed a feature's first stage, but only 28.5% passed when the whole feature was tested on the final code. Even the strongest agent finished under half the features it started.

The setup is close to how teams already work with agents^[2]:

• 73 features from the histories of 53 real repositories, in Python, Go, TypeScript and Rust.
• Ten agents, including Claude Opus 5.5, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.7 and DeepSeek V4 Pro.
• One working session per feature. The agent got each stage's specification in turn and extended its own code.
• No outside test results and no rollback. Nobody told the agent whether a stage had passed.

The strongest agent, Claude Opus 5.5, passed 83.6% of first stages and completed 49.3% of features. Features with five or more stages were almost never completed.

The test is strict on purpose. A team with a working test suite in continuous integration would catch some of these failures between stages, so real numbers will be better. What the benchmark isolates is how far an agent gets when nobody checks its earlier work, which is the condition a merge-rate metric assumes is fine.

Why do AI coding agents miss their own earlier bugs?

When an agent failed a stage, the next stage usually failed too, because it was built on the broken code. In FeatureArc, most failed later stages carried a defect from earlier, and in 94.9% of the stages after a failure, the agent never noticed the earlier error.

The numbers stack up quickly^[2]. After a stage that passed, agents passed the next one 70.2% of the time. After a stage that failed, that fell to 22.6%. Inherited defects showed up in 72.9% of failed later stages.

Later changes rarely broke earlier work: that happened in only 3.3% of features whose new behaviour worked. The trouble ran the other way. An early stage that never worked stayed broken, and everything after it was stacked on top.

The agents' own tests did not catch it.

A false green is a self-test that passes while the stage actually fails. Those rose from 28.7% after a good stage to 75.5% after a bad one. The agent was most confident exactly when it was most wrong, which is the same pattern we described as confidence running ahead of quality.

Two cautions belong next to those numbers. The "inherited defect" and "never noticed" labels come from three AI judges, one of which is also an agent under test; a person's blind review agreed with them on 56 of 60 sampled stages. And the paper is a preprint, not yet peer reviewed.

A second study points the same way at the merge itself. Researchers at the University of Arizona re-ran 30 merged performance fixes written by coding agents under controlled benchmarks^[3]. Eighteen delivered what they claimed, nine showed no real gain or made things slower, and 14 changed behaviour on inputs the tests never checked.

How do you test AI generated code at the feature level?

Test the feature, not just the change. Keep the acceptance tests from every stage of a feature, the checks that say what each stage must do, and run all of them after every new change. A defect from stage one then cannot hide behind green tests in stage three.

Three habits follow from what FeatureArc measured:

1. Keep every stage's tests live. A stage's acceptance tests do not retire when it merges. They run on every later change in the feature.
2. Do not build on red. If an earlier stage's tests fail, fixing them comes before the next stage starts. The 22.6% figure is what building on red costs.
3. Own the tests the agent did not write. An agent's green run after a failure is weak evidence. The acceptance tests should come from the person who knows what the feature is for.

FeatureArc did not test any of these fixes. They are what its measurements point to, not a measured result.

That third habit is where the human work sits. Agents bring speed and breadth to each change. People bring the judgment about what "done" means for the feature, and an acceptance test is that judgment written down where an agent can run it.

What should teams measure instead of pull requests?

Measure finished features. Track how many features pass their full acceptance tests on the final code after the agent's last change, and how often a new stage gets built on one that never passed. Those numbers can fall while pull request counts rise, which is why they are worth watching.

A short scorecard does the job:

• Features done: share of features passing every stage's tests on the final code.
• Built on red: share of stages started while an earlier stage was still failing.
• Feature lead time: days from a feature's first pull request to all of its tests passing.

None of this argues for slowing agents down. GitHub's numbers show how much work agents now start, and the speed is worth keeping.

A merged change is one step of a feature, and somebody, or a test suite they own, still has to check that the steps add up to software that works. That is the same bar we use for whether an app is ready for real customers: it works end to end, not one change at a time.