AI Security Patches Pass or Fail on Your Definition of Done

On August 6, 1Password's research unit, Off-by-1 Labs, reported that two frontier coding agents produced a clean fix for a security bug 26.0 percent of the time, across 6,080 graded patches on six recently disclosed vulnerabilities.^[1]

On September 15, Trail of Bits reanalyzed the published data and found that 2,634 of 3,067 patches, or 86 percent, blocked the supplied exploit.^[3]

Same models, same bugs, same data, and a public argument about which number is true.

A definition of done is the written, checkable statement of what has to be true before a piece of work can stop.

The two numbers are what you get when you change that statement and hold everything else still. The agents did not move between August and September. The finish line did, and the finish line was never the agent's to choose.

Can AI fix security vulnerabilities?

Sometimes fully, often partly, and the number depends on what you told it done means. 1Password's 26.0 percent counts a patch as clean only if it closed every exploitable path without changing other behavior, across every trial. Trail of Bits' 86 percent counts whether the supplied exploit stopped working, where the agent could test.

1Password's study was serious work: three modes (a one-shot arm with no shell access, an iterative arm with a reproducer script and up to ten runs, an exploratory arm with no reproducer at all) and nine prompt templates that vary how much the agent is told and whether the fix direction is right.^[1]

The top grade requires full remediation with no erroneous change to application behavior.

The blog post's headline: patches that did not resolve the vulnerability, added a new one, or both, 53.9 percent of the time.^[2]

Trail of Bits read the same data differently. The firm co-authors patches with agents through its Patch the Planet program with OpenAI and sells AI-augmented security work, which is worth knowing and does not change the arithmetic:

• Two of the nine prompts tell the agent to apply the wrong fix. They account for 22 percent of the trials.
• One of the three modes prevents the agent from building or running code. It accounts for 36 percent.
• The six bugs were chosen because their fixes were hard. Clean-fix rates ranged from 3 to 60 percent across them.

Remove the misdirected trials and the blind ones, count whether the exploit is blocked, and the number is 86 percent. Trail of Bits attaches its own caveat in the same paragraph: "Blocking that exploit does not establish a complete repair."^[3]

Why did 1Password and Trail of Bits get different numbers?

Because both measured honestly against different finish lines. The 26 percent is what you get when done means every path closed and nothing else changed, averaged over trials where some agents worked blind and some were pointed the wrong way. The 86 percent is what you get when done means the exploit no longer works and the agent could check.

The evidence that an agent runs to whatever finish line it is handed is in 1Password's own report. On the Gemini CLI bug, the models routinely patched the visible bypass and left open a second path that loads environment files. That signature appears in 29 percent of one model's one-shot patches and 64 percent of the other's.^[1]

Give the iterative arm a reproducer that exercises the second path, and the signature falls to zero.

1Password names the mechanism plainly: the models "patch to the specific bug they are shown and the test they are given, rather than to the safety of the code they are editing." It calls the narrow fix "a form of reward hacking," the behavior we wrote up when a model can game the benchmark. Then it measures the mechanism.

Aim, measured

Correct guidance on how to fix the bug produced a 65.0 percent fix rate. No guidance produced 50.4 percent. Confidently wrong guidance produced 15.2 percent. Those rates count fixes that changed behavior along the way, so they sit above the 26.0 percent headline, and the ordering is the point.

A bare nudge of a prompt scored 51.8 percent and a fully specified brief scored 76.3 percent. In the report's words, "the features of the bug and codebase appear to matter significantly more than the model used for patching."^[1]

AI amplifies your direction, right or wrong, and here the direction moved the outcome more than the choice of model did.

What is a definition of done for an AI coding agent?

A written, checkable statement of what must be true before the agent may stop. Anything you leave out of that statement, the agent is free to leave out of the fix. For a security patch it has four parts:

• The test that failed before the change passes after it.
• At least one other path to the same failure is covered.
• Existing behavior is unchanged.
• The evidence is saved for whoever reviews the work.

Even the people running the benchmark had to write theirs twice.

1Password's manual review found its automated graders treating a bug as fully fixed whenever the reproducer stopped reproducing it, which is one input, not a class of inputs. The rubric was corrected mid-study so that "Fixed" meant "fixed across all code paths addressed by the upstream patch."^[1]

After the correction, the automated grade matched human review on the exact outcome 65.9 percent of the time, and the two grader models disagreed on 36.8 percent of the same patches.

If a security research team needed two passes to define done for six bugs, expect to write yours once.

We have argued before that a check proves only what it was written to prove, and a proof-of-concept exploit is the narrowest check there is.

The gap is not a security-only finding

A separate group measured the same shape on ordinary repairs. SWE-Gate, published September 3, built 303 repair tasks from 75 open-source Python repositories, each with a functional test and a second, review-derived constraint taken from what a human reviewer actually asked for on that kind of change.^[4]

Under one scaffold, four models produced 644 repairs that passed the functional tests. Of those, 221, or 34.3 percent, failed the reviewer's constraint.

1Password's own fragile flag, a self-reported annotation for fixes that block the exploit path but leave the vulnerable code reachable from elsewhere, sits at 37.5 percent of successful fixes. Two groups, two kinds of work, one shape: about one in three passes is not done by the standard a person would apply. Neither paper cites the other.

The benchmark result is also a property of the stack it ran in, which the research already says in general.

A same-day audit of the SWE-bench leaderboard found within-model scaffold ranges of 29.8 points against an 8.8-point spread among the top thirty entries, in an observational design that does not identify causal effects.^[6] The patching dispute is the concrete case: the stack included whether the agent could run tests and what the grader called fixed.

Can you automate definition-of-done checks?

Partly, and the gate matters more than the volume of review. What Trail of Bits shipped alongside its rebuttal is a definition of done in executable form: a post-patch validation skill that makes the agent prove the bug before it claims the fix.

The skill then makes the agent test another path to the same failure, check for regressions with project tests, a sanitizer, or a bounded fuzzing run, and treat broken test runs as inconclusive rather than as passes.^[3] Trail of Bits notes it was not used in its own earlier patching work. That is the finish line written down.

It is the same shape as Proof-or-Stop's result from July. Gating "tested" and "done" on mechanical evidence produced zero false-done results across ten unattended loops and cut visible-pass, hidden-fail results from 31 to 2 per 1,800, with the gain "associated with enforcing review as a lifecycle gate rather than merely adding a reviewer."^[5]

It is the conclusion we reached about the gate, not the job title: the check has to sit where the work cannot pass without it.

The human baseline, with its conditions

Trail of Bits reviewed the first fixes for 2,265 vulnerabilities across 236 of its own assessments and found 283, or one in eight, failed to fully resolve the issue.^[3]

Those came from maintainers who had a detailed report and knew a review was coming. On one bug, a maintainer and an agent working separately made the same mistake.

The firm's own line is that a fair comparison needs the same tasks and conditions. The claim here is not that agents beat people. Nobody, human or agent, gets to skip the definition.

Who owns the definition of done when an agent writes the code?

The builder does. The model will not choose it, and the benchmark will not choose it for you.

The two firms arguing about the benchmark converge on that point from opposite sides: 1Password's remedy is turning single proofs-of-concept into thorough test suites, and Trail of Bits' remedy is a skill that writes the second test before the agent may stop. Both are the same instruction, aimed at the builder.

In the Hacker News thread, Trail of Bits' CEO called it "kind of insane that anyone would think AI can only patch things correctly 26% of the time," and another reader replied that two graders disagreeing on 37 percent of patches means "there's no actual methodology."^[7] Both are arguing about which number is true.

Neither number is a fact about the agent.

When you hand an agent a bug, you are also handing it a finish line, whether or not you wrote one.

The one it infers from a single proof-of-concept is "make this input stop working." The one you want is "close the class, keep the behavior, show me the evidence." The gap between those two sentences is where 26 percent and 86 percent both live.

Two days ago we wrote that a managed agent leaves you three jobs: define done, set the fence, check the work. This is what the first of those jobs looks like when a security team measures it, and it is the part of the work that stayed yours.