AI Legacy Code Modernization Breaks More Working Code Than It Fixes

Cisco's president has handed the company's legacy code to one 27-year-old engineer, Padmanabhan Krishnamurthy, with an unlimited AI budget. "I've given him unlimited tokens because he's going to refactor code," Jeetu Patel told Fortune at a roundtable dinner this week. "So I'm like, you can go, you can go nuts."^[1]

The goal is 98% of Cisco's legacy code within 24 months, which he reckons two people can do.

Fortune reports no results yet, no language and no plan for checking the new code. That last gap is what this post is about. The top comments on Hacker News got there first: rewrite it in service of what, and does anyone on the project know why the old code was written the way it was?^[2]

The day before Fortune's story, five researchers in Bangladesh posted a study that asks a small, measurable version of the same question. Their answer was that every AI model they tested left fewer working files than it started with.

Can AI modernize legacy code without breaking it?

Not reliably, in what its authors call the first study to score AI rewrites against leaving the code alone. Five AI models modernized 100 legacy files from WordPress, the website software. Untouched, 83 files passed every behavior check. After the best model's rewrite, 59 did. Every model ended with fewer working files.^[3]

The setup was a shortcut many teams already take: paste one old file into a chat assistant and ask for current PHP (the language WordPress is written in). The instruction asked for modern code and identical behavior, down to the side effects and the code calling it.

The researchers then ran 630 hidden checks, tests the models never saw, on the originals and on every rewrite. They also ran Rector, a rule-based upgrade tool that applies fixed rewrite rules, the way a developer could without any AI at all.

The sample is narrow, and the authors say so.

The models were 2025 releases (Claude Sonnet 4, Gemini 2.5 Flash and Pro, GPT-5 Codex and Llama 3.3), each given one file, one prompt and no chance to run its own output. Newer agents that test as they go would do better, and they would need the tests too.

How many working files did each model break?

Each AI model broke between 32 and 51 working files, while fixing at most 10 that had been failing. Rector, the rule-based tool, broke seven and fixed six. Every model lost at least three working files for each one it repaired, and only a comparison with the untouched files shows it.

Rector has a home advantage, which the paper flags. The same tool picked the 100 files and defined what counted as outdated, so the sample favors the patterns it already knows. That cannot explain the model losses, though, because most of them happened in files that needed no change at all.

Why did most of the breakage come from changes nobody needed?

The AI models changed far more than the new language version required. Rector flagged 511 outdated spots across the 100 files, and only 37 of them (7%) had to change to run on current PHP. In working files that needed no required change, the models broke between 36% and 60%. Rector broke 8%.

Optional changes made up 82% to 87% of each model's updates, the parts that make code look current.

The damage was rarely subtle. Most failed checks came from files that no longer loaded at all, often because of a type declaration (a label stating what kind of value a function takes or returns) that PHP itself refuses. The paper's starkest example:

// Original (also retained by Rector)
public function __set($name, $value) {
    return $this->$name = $value;
}
 
// Claude Sonnet 4
public function __set(string $name, mixed $value): mixed {
    return $this->$name = $value;
}

The original passed all nine of its checks. PHP requires that particular method to return nothing, so the rewritten file fails to load and all nine checks fail. "Avoiding this error requires knowledge of PHP, not of WordPress," the authors write.^[3] No missing project context explains it.

Other regressions changed what callers depend on: a number returned where text was, public fields made private, classes renamed.

Why isn't a modernization score enough?

A modernization score measures how current the code looks, not whether it still works. In the study, the model that modernized the most (72.8% of flagged spots) ended with 55 working files, fewer than two models that modernized less. A rewrite that reports only its modernization score is reporting the wrong number.

The authors put the rule in one sentence: "A modernized file is an improvement only if it beats the original."^[3] Their recommendation follows from it. Measure against a no-change baseline, and report modernization and behavior preservation as two separate numbers, because a single blended score hides exactly the losses above.

The baseline for any rewrite is the code you already have, and it usually works better than it looks.

What does a legacy rewrite need before the first token?

A referee: the old code's behavior written down as tests, run against every change. GitHub's agent-led Rust port of its Copilot runtime also had one engineer in charge, and it worked because the existing end-to-end tests ran at every step. A change that failed one did not land.

When that port shipped, the lesson was that the old code was the referee. A month later the same pattern explained why agents finished game ports and stalled on a Photoshop clone: the oracle, not the spec. The PHP study shows the same work with the referee removed.

Microsoft, which runs agents across the Windows and Azure codebases, describes migrating feature by feature, with tests that compare old and new behavior and tests held back from the agent so it cannot tune itself to them.^[4] Two of its AI engineering vice presidents open the piece this way.

None of this says the Cisco project will fail. Nothing about it is public yet beyond the goal and the budget. It says which number will decide it: working behavior kept, measured against the code Cisco already has.

What should you do before giving a rewrite unlimited tokens?

Write down what the old code does and check the untouched code against it first, so you have a baseline. Then require a reason for every change, keep mechanical upgrades on rule-based tools, and report files broken and files fixed beside any modernization score.

The order matters more than the tooling:

1. Capture current behavior as tests, including the odd behavior callers rely on. The PHP study's checks encoded what the legacy code did, not what it should have done.
2. Run them on the untouched code. That pass rate is your baseline, and it will likely be higher than the code's age suggests (84% of checks passed in the study).
3. Require a reason for each change. "Required by the upgrade" and "looks newer" are different budgets.
4. Use rule-based tools for the mechanical part. Here the rule-based tool broke seven files where the models broke 32 to 51.
5. Report broken and fixed counts with every modernization score. A rewrite that improves one and worsens the other has not improved.

Even the cheapest gate pays. A simple syntax check that kept the original whenever the rewrite would not parse recovered 8 to 15 files per model. It still left the best model 14 files short of doing nothing, because a file that loads can still return the wrong thing.

Token budgets are the easy part of the plan to change, and unlimited token budgets are already normal at the labs. What a budget buys is attempts. What a rewrite is worth is its cost per finished task, and for legacy code a finished task means the new file still does everything the old one did.

Unlimited tokens remove the cost of writing new code. They do nothing about the cost of knowing it still works, and on a legacy rewrite that cost is most of the job.