AI Slop in Open Source Has a Measurable Form: Skipped Documentation

On October 20 the Linux Foundation published a survey in which maintainers estimate that 25% of incoming pull requests are low-quality "AI slop," rising to 31% at large projects.^[1] The same week, a Ruby maintainer archived every one of his gems, Lobsters retired its vibecoding tag, and the Linux kernel dropped a disclosure tag almost nobody used.

The quarter is an estimate. The day before the survey landed, two researchers at Monash University published a count of what agent pull requests actually do, and one number stands out: 63.2% of them change code without touching a single documentation file. That is a property of the pull request, so a project can require it.

What did the Linux Foundation actually find?

The survey reports that 59% of maintainers see growing contribution volume since generative AI arrived, and that maintainers estimate a quarter of incoming pull requests are low-quality AI output. It also reports that 53% of maintainers disagree that generative AI has improved community trust in their projects.^[1]

The rest of the picture is mixed. Only 35% of projects have a formal policy permitting AI use, and 32% have no policy and no plans for one. Among contributors, 61% say they review and fully test AI-generated code before submitting it. The press release gives no sample size or fieldwork dates.

The week around it filled in the mood:

• Paul Battley archived his Ruby gems and closed with "You want changes? Go ask your agent or something."^[2]
• A Lobsters moderator marked the vibecoding tag inactive after a run of moderation fights around it.^[3]
• The kernel retired its Assisted-by tag after barely 10% of AI-assisted patches carried it, according to coverage of the mailing list, and Linus Torvalds called bans on AI-assisted code "idiotic."^[4]

The prevailing read follows: AI is flooding the commons with plausible junk, review time is scarce, and the reviewers are leaving. A July paper backed it with data, finding merge rates for one-time contributors fell 18.18% against the counterfactual as volume rose.^[5]

The Linux Foundation's recommendation is the right one. Hilary Carter, its SVP of Research, put it as "setting clear expectations for GenAI, putting responsibility for testing and disclosure on contributors and giving maintainers better tools to manage incoming pull requests."^[1] Our disagreement is about what to count.

Can you measure AI slop in a pull request?

Not as "slop." The survey asked maintainers how much of what arrives is low-quality and they said about a quarter, which is a true report of a feeling. But slop is not something a pull request has or lacks, so no contributing guide can require its absence. What a pull request changes, and what it leaves alone, can be counted.

That is what Will De Michele and Kla Tantithamthavorn did. Their paper, accepted at the APSEC 2026 conference, reads 29,669 pull requests written by five coding agents (Claude Code, Cursor, Devin, GitHub Copilot and OpenAI Codex) across 2,585 repositories, and asks one question of each: when the agent changed code, did it change any documentation in the same pull request?^[6]

Documentation here means Markdown and text files, anything under a docs folder, and the README, LICENSE and CHANGELOG. The label records whether any of those changed, not whether the change was good. Inline comments don't count. About two-thirds of the pull requests come from Codex, and there is no human baseline to compare against.

What do agent pull requests skip?

Most of the time, the agent changed the code and none of the documentation. Across all 29,669 pull requests, 18,751 (63.2%) changed code with no documentation update. The rate depends on the job: 55.2% of feature work went undocumented, and 77.5% of refactors did. Every agent did it, from 48.1% of pull requests for Claude Code to 72.8% for Devin.^[6]

Some internal fixes and refactors need no documentation change, so part of that gap is legitimate. But the pattern holds inside every agent: feature pull requests carried documentation 1.6 to 2.1 times as often as fixes.

Agents document what they add, and far less of what they repair or restructure.

Does skipping the docs cost anything later?

Yes, though the raw numbers hide it. Pull requests that updated documentation drew more later fixes, but only because they were 3.9 times larger. Compared file by file, a code file changed without documentation was followed by a fix to that same file 8.6% more often at 30 days and 11.7% more often at 90.^[6]

That size trap is the mistake a team would make with its own dashboard. Documented pull requests touched about 25 files on average and undocumented ones about six, so the bigger pull requests offered more places for a later fix to land.

In absolute terms the gap is one to two percentage points per file, and the attribution is by timing, not cause. Across thousands of files changing every week, that is a steady extra trickle of rework.

Documented changes also merged faster despite their size (median 40 seconds against 184), though merging here is mostly automated.

Why does the skip compound?

Because nobody comes back for it. Within 90 days, a file changed without documentation was later re-documented 24.0% of the time, against 48.2% for a file changed with documentation. Holding change size and agent constant, a documented file had 3.2 times the odds of being documented again, and the effect survived comparing files inside the same repository.^[6]

This is the mechanism behind a pattern we described in September, why AI-built code gets harder to change every week. The rot has a specific form: one omission, repeated at agent volume and repaid at half the rate it was taken on.

Is documentation the only change agents skip?

It is the only one measured this precisely, and the neighbors point the same way. Deakin researchers followed 6,774 merged agent pull requests and found they drew verified follow-up fixes at 1.62 times the odds of human merges in the same repositories, with 69.6% of those fixes coming from the same agent.^[7]

A July benchmark of 106 issues across 49 repositories found that today's agents "almost never proactively retrieve the contribution rules," and never refused to contribute where AI was banned.^[8] We covered that in July as agents do not read the rules. On GitHub, an agent can now open a pull request that gets no human review at all.

Put the three findings together and one pattern shows. The agent does what the check checks. Tests and merge gates look at the code, so the code gets attention. Nothing looks at the README, the CHANGELOG, or the paragraph in the contributing guide, so those get skipped.

What should a contributing guide require?

Properties of the diff that a bot can check before any person reads the pull request. The measured one comes first: when behavior changes, the documentation changes in the same pull request, and the pull request doesn't merge until it does. An agent can't skip a failing check the way it skips a sentence.

The rest follow the same logic, without the same evidence yet:

• Documentation updated in the same pull request when behavior changes (measured)
• A test that fails before the change and passes after it
• A disclosure line saying an agent wrote or assisted the change
• A short note on what was deliberately left alone

This is not an argument for a ban. In August we wrote that the evidence says regulate, do not prohibit: across 29,624 repositories, policies built on transparency and responsibility did better than restrictive ones.^[9]

A ban is a sentence the agent never reads, and a slop filter is a classifier arguing with a maintainer's feeling. A required change is either in the diff or missing from it.

The Monash authors suggest the same direction from their own data: treat drift as a risk signal, and ask agent developers to document files they revisit, not only files they create.

What about the maintainers who already left?

Not all of them left over pull request quality, and a docs check won't bring them back. Battley's reasons are about who profits from the commons: "I no longer feel inspired to contribute to a commons that's being strip-mined by massive corporations to feed the very machines whose unthinking output pollutes it."^[2] That stands on its own.

Peter Eisentraut, a PostgreSQL committer, describes the other side: eight to ten significant features pulled from PostgreSQL 19 after beta, where one per cycle would be normal. AI-assisted review now finds old defects fast, so "the effective quality bar has been raised after the time the feature was written and committed."^[10]

He adds that this is "not due to 'vibe coding.'" We covered the reverts last week.

The survey's mentorship number sits between them: 46% of maintainers say generative AI increases the need to mentor new contributors, against 32% who say it decreases it.^[1] An agent learns nothing from a rejected pull request, but it does respond to a failing check.

What does this mean for a team running agents on its own code?

The same gradient applies inside a company. Refactors are where documentation gets skipped most, and the review that approves them is looking at the code. Six months later, that file is the one nobody on the team can explain.

Put the documentation change in the merge check rather than a guideline, and track the re-documentation rate the way the Monash team did.

Designing for it means writing down what a contribution must carry, in terms the contributor can't skip. The first line is already measured: when the code changes, the documentation changes with it.