How to Verify AI Vulnerability Scanning Results Before You Count Them

OpenAI's security pipeline found 948 distinct ways to take over a Linux machine in one mid-October scan, wrote a working attack for each, and did it for about $1.2 million in API credits.^[1] That was the number people repeated last week.

The line most coverage skipped sits further down the same report: an early version of the pipeline counted attacks that weren't what they claimed to be.

That second line matters more for anyone who reports what their agents did, because it describes a problem with the count, not with the model.

What did OpenAI's agents actually count?

OpenAI's agents each got one assigned kernel bug and were scored on whether they reached root, full control of the machine. In an early run, human review found that 20 of 200 reported successes reached root through a different, unreported bug. The check confirmed the outcome without confirming how the agent got there.

OpenAI's words: "This suggests that even successful exploitation by agents needs proper validation."^[1] The agents did get root. They just didn't do the task they were being scored on, and the score couldn't tell.

What OpenAI did next is the part worth copying. It tightened the task and added an independent check that each final attack used its assigned bug.^[1] Then it wrote the lesson down as a rule:

How many Project Glasswing findings were fixed?

VulnCheck counts 217 fixed out of 37,966 findings in Anthropic's public list for Project Glasswing, its program to find and report open-source bugs with its models. Only 7,349 have been made public, and more are marked duplicate or withdrawn than fixed.^[2]

VulnCheck tracks 1,654 findings marked duplicate, nearly eight times the number fixed, and 419 withdrawn. Over the same months, Anthropic's own reported share of real findings rose from 90.8% to 94.2%.^[2]

VulnCheck sells vulnerability data and argues the launch was oversold. It also writes that AI is a valuable tool for finding vulnerabilities, and its counts stand on their own.

Where did the check go?

Curl shows where the checking went. Curl, the widely used file-transfer tool, received a batch of 45 reports from Anthropic's new self-service scanner, and its maintainer is publishing them in the open. Of the 34 published so far, 19 are marked not a vulnerability, six are confirmed low severity and nine are still waiting.^[2]

Anthropic is upfront about this. Its scanner's reports are "fully model-generated, without human review or triage," and Anthropic has said it is "bottlenecked on our human capacity to validate these findings."^[2] The check still happens. It moved downstream, to the people receiving the reports.

One outside count shows the same gap from the other side. OpenAI says its new security model found 81 high-severity Firefox flaws. A developer who parsed Mozilla's advisories through November 3 found 41 credited to OpenAI.^[3]

He says plainly that the rest may land in later advisories. That's the point: from outside, nobody can yet tell a confirmed count from a reported one.

Why is a passing check not the same as a result?

A check passes when the outcome it looks for shows up, whether or not the work produced it. Something else can supply the outcome: a second bug, a library default, a forgiving test setup. A count of passes then counts the check's blind spots along with the real work.

The two readings going around this week both skip that step. One says AI now finds bugs faster than anyone can fix them, which takes the count as true and argues about capacity. The other says the counts are hype, which takes them as false and argues about marketing. Both treat the count as a fact about the model.

It's a fact about the check, and a new study measures exactly how.

The same code, tested twice

Three researchers at Bangladesh University of Engineering and Technology had ten AI models write 9,100 working solutions to 24 everyday backend tasks. They then crashed each solution at every point it wrote to the database, re-ran it, and checked whether the data came out right.^[4]

They did it twice, under two common settings of Python's built-in database library. By default, the library groups a function's writes so they land together or not at all. In the second setting, called autocommit, each write saves the moment it happens, the way Django runs it.

Of the database code that survived every crash under the default setting, 54% was left half-updated under autocommit.^[4] That held for every model. Claude Sonnet 5, the safest in the default setting, went from 7% unsafe to 31%.^[4]

Nothing about the code changed between the two runs. The first test was partly measuring the library. The authors' rule for telling the difference: "Code that passes both earned its pass; code that passes only one borrowed it."^[4] They also say the question is what a passing test credits, not whether models write worse code than people.

One failure, found twice

OpenAI's wrong-bug attacks and the study's borrowed passes are the same failure. In both, the check confirmed an outcome (root access, intact data) that something other than the work produced. It's the shape of a model that can game the check, except nobody here was gaming anything.

The fix is the same too. Add a second check that would fail if the credit were borrowed: the attack must use its assigned bug, and the code must survive the less forgiving setting. The study is about ordinary database code, not security, and the mechanism is what carries over.

How do you verify an AI-found vulnerability?

Verify how each finding was reached, not just that a result appeared. Confirm the attack or test hits the specific flaw reported, rerun it under a second, stricter setup, and have someone other than the finding agent try to disprove it. Count only what survives.

In practice that means a short list of checks, each one asking about the path:

• The attack or test exercises the exact line or flaw named in the report.
• It still works under a stricter setup than the one it was found in.
• It isn't a duplicate of something already reported or fixed.
• Someone other than the agent that found it has tried to break the claim.
• The fix is confirmed by a test that fails without it.

That last item is the same question we asked about what counts as done for an AI patch. A finding isn't done when the agent reports it, and a patch isn't done when the tests go green.

What should a team count instead?

Count what a second check confirmed, and report it next to what was claimed. Every agent metric has the same shape as these bug counts: tests passed, pull requests merged, tickets closed, bugs found. Each is a count of what some check accepted.

For each number you report about your agents, ask two questions. What does the check actually confirm? Could a result pass for a reason other than the work? Then add one check that asks how the result was reached, and run it before the next step builds on it.

Where does the evidence stop?

The limits are real. OpenAI's counting error was in an early run, and OpenAI fixed it. VulnCheck is a vendor with a view, curl is one project, and its batch isn't finished. The study covers one library setting in one language, on tasks the authors wrote.

The direction holds anyway: a count with no check on the path is a count of what the check let through.

The labs that found these bugs also had to build the second check. Anyone who reports what their agents did will have to build one too, and the honest report names which check produced the number.