How to Verify AI-Generated Code Before the Next Request Builds on It

Revolut moved its engineering organization, roughly 2,000 engineers, onto a self-hosted open model this year and cut its AI coding bill by about 90 percent.^[1] The savings are real. The argument since has been about one question: is a cheap open model good enough, or are the frontier models still worth the premium?^[2]

A study two Amazon researchers published two days earlier measures something that question skips. When a coding agent works through a sequence of requests alone, each built on the last, about half the sequences end up correct.^[3]

What decided the outcome was whether anyone checked the work before the next request built on it, far more than who wrote the code.

What did Revolut actually change?

Revolut made GLM-5.3, an open-weight model it runs on 56 of its own Nvidia B200 chips, its default for AI coding in place of paid services, at about a tenth of the cost of Codex or Claude Code. It reported faster coding time and lead time. The measures it reported were cost and speed.^[1]

The company started on GLM-5.1, moved to GLM-5.3, and still routes some work, such as interface work, to other models. Before the switch, tokens were rationed. David Abitbol, who leads ops and infrastructure there, said engineers were spending "$1,000 a month, $10,000 a month with not so much ROI."

Cheap tokens ended the rationing, and Abitbol describes how the work changed:

That last line describes unattended work. An engineer hands off a task and the agent reports back when it thinks it is done. Whether the result holds up was not part of the story, and the article does not say how Revolut measures it.

Is AI-generated code reliable over a sequence of requests?

Request by request, mostly yes. Over a sequence, much less. In Amazon's BOBA study, ten coding agents completed about 95 percent of requests started from checked code, but only 51 percent of 20-request chains in which each request was built on their own unchecked work.

What the study measured

The paper is called Built on Broken Assumptions, BOBA for short, by Alejandro Cuadron and Arpit Gupta. They built 280 chains of 20 requests from the histories of 140 real open-source projects. Each request ran on whatever the agent left behind, and hidden tests the agent never saw judged the result.^[3]

Inside those chains, the agents still got 66 to 84 percent of single requests right.

The study has clear limits. It covers Python only, many of the projects are small, and the hidden tests check fixed behaviour rather than what a user meant. Models wrote and audited the specifications, which no human has yet reviewed. It is a preprint from two authors at Amazon.

The result that matters most for a team choosing a model is about authorship. In the authors' words, "Worse code is not the cause. Once checked, agent code is nearly as good a base for later requests as human code. The cause is the mistakes agents miss."^[3]

Why don't the agent's own tests catch the mistakes?

Because the agent tests what it wrote, not what was asked. In the BOBA study, agents ran their own tests before almost every submission, and 59 percent of the mistakes they missed came right after a clean local run. A green result is what convinced them the request was done.

The numbers behind that are blunt. Agents ran a test before 97 percent of submissions, and a passing run barely changed the odds of a miss: 8.3 percent of submissions after a green run failed the hidden tests, against 8.6 percent overall.^[3]

Then the miss compounds. A request issued on a broken repository fails about twice as often as one on a clean repository, and the odds of failing roughly double with each additional unchecked mistake beneath it. Only 14 to 33 percent of failing tests were ever repaired by the agent's later work.

A separate study saw the same thing: agents pass the first change and fail the feature.

Can another AI model review the code?

Not reliably as the only check. Asked whether an implementation was correct, the ten models were right 53 percent of the time. Lenient reviewers, GLM-5.3 among them, accepted 77 to 95 percent of wrong implementations. Strict reviewers rejected 41 to 58 percent of correct ones.

All four Claude models were in the lenient group too, so this is not a point about cheap models.

A second agent that reads the finished code has the same blind spot as the first: it sees what was built and has to guess what was asked. The paper's own conclusion is that a pipeline relying on one agent to review another "does not stop models building on broken assumptions."^[3]

We have argued before that quality depends on who checks the output. This is the measured version of that, and the checker that works is one that runs the code against what was asked.

Does the cheaper model matter?

Some, but the outside check helped every model. One external "verification failed" message after a missed request, plus one repair round, lifted average chain completion from 51 to 59 percent. GLM-5.3, the model Revolut runs, gained the most of the ten under the paper's benchmark rules, 22 points.

The message was a single sentence saying the previous submission failed. It named nothing. Told only that, the agents fixed part of their misses in one round, and a repaired mistake stopped hurting the requests that followed.^[3]

The gain varied widely, from 1 to 22 points depending on the model, and the repair round had a price: it raised the share of chains where a passing test later broke by 3 to 9 points.

The cost math still points the same way. Price per token is not price per finished task, and a cheaper model that finishes more chains because someone checks its work is a better deal than its token price alone suggests.

What cheap tokens change

Cheap tokens also change how much unattended work a team runs. Abitbol said the biggest gains came from handing models debugging and ticket work they used to save tokens for.^[1] We saw the same pattern in the Cisco rewrite numbers: unlimited tokens buy more changes, not better code.

More requests per engineer means more unchecked foundations, unless the check scales with them.

How do you verify AI-generated code between requests?

Check each request from outside the agent before the next one builds on it. Run the feature or a test suite the agent did not write, send any failure back in plain words, and count how many sequences of work end up correct, not how fast each task finished.

The paper's advice to users fits in four moves:

• Check before you build on it. "If a request is the foundation of the next, run it or try the feature before issuing the next request rather than after."^[3] Checking afterward is harder, because the next request has already landed on the mistake.
• Say what you saw. Naming the failing test once did what repeating "that's wrong" took about three rounds to do.
• Keep model review as a filter. A reviewer agent can flag likely problems, but at 53 percent it cannot be the gate.
• Measure finished sequences. Coding time and lead time show how fast work moves; the share of chains that end correct shows whether it held.

For a team running agents in parallel, the practical change is small. The check moves from the end of the project to the moment between one task and the next, and it has to be something the agent did not write.

Revolut chose its model on cost, and on that measure the choice looks right. Reliability is a separate decision, made in the minutes between one request and the next, and it goes the same way whichever model writes the code.