A client recently asked how long it would take to clear the static analysis backlog in their codebase. They'd done the math themselves and come up with three months. I told them I couldn't give them a number. If they pushed me, I'd say a year, and I'd hold that loosely, because the thing that decides the answer isn't steady enough to plan around.
This article is about the gap between three months and a year. Almost none of it comes down to how fast an AI model writes code.
The project
The codebase is millions of lines. The static analyzer reports warnings in the six figures. Most aren't critical bugs. They're the edge cases and possible security holes that never show up in normal use or normal testing, and never go away either.
We built a tool to work through them. It pulls findings from the analyzer, hands each one to a coding agent, checks that the change removes the warning without creating new ones, confirms the code still builds, and opens a pull request. There are more moving parts than that makes it sound, and every one of them can break. I haven't written a line of that tool's code. The agents wrote all of it from scratch. My job has been to design it and review it, not to program it.
The fixes are small on purpose. On a codebase this size, the client doesn't want clever. They want one line, sometimes five, an added conditional and nothing else. We're expected to keep changes minimal, because piling more code into a codebase this big does its own kind of damage.
So generation isn't the hard part. An agent writes a correct minimal fix in minutes, and it'll do that all day without getting bored, which is more than I can say for a person doing the same job. That makes the math look easy. Fixed backlog, known rate, divide. Three months is a fair answer to that sum. The client wasn't being naive. They were doing the right math on the wrong number.
What refactoring needs
It helps to be clear about what this work involves, because that's what explains where the real limit sits.
Refactoring means changing how code is built without changing what it does. That sounds mild. It isn't. Every change has to be measured against something the code can't tell you: what the system is meant to do, rather than what it happens to do now. So you set a boundary, a line the agent isn't allowed to cross. For a library or a module, that's usually the public API. For a whole application, you pick the outside interfaces, a REST API or a CLI, and treat what they do as the reference point before and after.
An agent can't see that standard on its own. You can hand it documentation, and documentation helps, but prose is slow to read, costs a lot of tokens, and leaves the model room to interpret its way into a regression.
Tests work differently. A test gives a clean pass or fail, and that's what turns an unpredictable tool into a process. The agent finds out right away whether it broke something. You get a signal you can build automation on, instead of reading every diff yourself. Everything I've learned with these tools says the same thing: the more of your intended behavior sits in tests, the better the output and the lower your costs later. Time spent on tests pays back in more ways than it costs.
You also have to protect the tests. Even when the prompt says clearly that tests must not change, an agent will sometimes try anyway, usually to get around a bug it can't fix. It's rare, but it's the kind of rare that messes everything up. So we keep plain scripts that check whether the files meant to stay the same actually did. Not an agent watching another agent. A script, because you can't talk a script into anything.
Where the time goes
Into the pipeline, and a lot of it.
Much of the time, the tests just fail. Not because the fix is wrong. They just fail. You restart them, they fail again, you restart a third or fourth time, and a day is gone getting one pull request merged. That isn't a small annoyance sitting next to the schedule. It is the schedule. Spread it across a six-figure backlog and the whole picture flips. What sets the timeline isn't how fast you can write changes. It's how many the system around the code will take per day.
Those are two different numbers, and on a job like this they can be ten times apart.
Tests are the enabler and the limit
This is the part I find most useful, and it's a little uncomfortable. Tests are why agentic refactoring works. Tests are also why it doesn't go faster. The same suite that lets an agent run without supervision is the one deciding how fast its work can land. You don't get to take the first half and skip the second.
Most teams have never had to notice, because the slow step used to be the person. When a human writes the changes, a pipeline that needs three or four tries to go green is annoying and nothing more. The retries disappear into that person's thinking time and never reach the schedule. Take the person out of the writing step, and the retries have nowhere left to hide. The pipeline stops being annoying and turns into the plan.
Here's how I'd put it. AI didn't add a new bottleneck to our process. It took away the one that was hiding the others.
Why you can't just let it run
A fair objection here is that we're being too careful. There are well-known stories of teams pointing a model at a big migration, walking away, and getting something impressive a week later. So why not set the goal, drop the guardrails, and come back when it's done?
Part of the answer is that the merge rate doesn't care who wrote the change. But there's a firmer limit underneath.
How much an agent can do on its own is capped by the context window. Today's agents can spawn sub-agents and run for a long time. I've had sessions go four or five hours, working through a task, reviewing their own output, fixing what they found, and coming back with results that genuinely surprised me. Even then, the session running the show has a ceiling. It can't hold a whole project in view. Compaction helps, and it still loses things that mattered.
So scoping stays my job. These tools are much better than me at finding things in code I don't know. They're not better than me at deciding what should change. What I do is write out what I'm after as a rough stream of thought, ask an agent to check it against the current code, and get back a scope. That scope often tells me the small change I had in mind touches half the project through some integration I'd forgotten. Then I ask for a plan, usually a list of small commits I can review, and we go back and forth. Sometimes five rounds. Sometimes thirty. Only when I believe the plan do I hand it to automation.
The way to get long unattended runs isn't more freedom for the agent. It's more scaffolding around it: ordinary Bash or Python scripts that carry what matters from one session to the next and add structure the agents can't hold on their own. Our automated review returns one of three answers: pass, pass with findings, or blocked. It'd be easy to wire that up so a blocked result sends the agent off to fix the blocker first. I've chosen not to. When something serious turns up, I'd rather stop and look at it myself, because sorting out a problem while it's small costs less than pulling apart everything built on top of it later.
Choosing how much supervision to buy
There's a related decision people skip, and it's really about staffing. How much freedom do you give the agent, and how much of an engineer's time does that cost you?
The tightest mode is strict and minimal. The boundary holds, and the agent is told to stop and report rather than cross it. Token costs stay low because nothing unnecessary happens, and risk stays low because you keep control. You pay in attention. The process spends a lot of time stopped, waiting for you.
The middle mode, and the one I use most, is controlled expansion. The agent can cross the boundary when it really has to, but it has to log every crossing with what changed and why. Two good things come out of that. The agent often thinks again about whether a change was needed, once it has to write down the reason. And when it goes ahead anyway, you get a record instead of a surprise. This buys real autonomy, because you can check in now and then instead of watching constantly. It also turns up genuine improvements in corners of the codebase you'd never have opened. What you trade is token cost and the risk of work you have to undo.
The loosest mode is what most people picture when they hear about AI writing code. Set a few architectural rules and let it go. It's useful for prototyping, for questions like whether swapping one library for another would even work, and it's cheap if you're running a small or local model. It won't give you production-ready code, and I wouldn't point an expensive model at it, because today's agents burn tokens faster than they return value with that little supervision.
Getting this choice wrong gets worse over time. If the unwanted work is fresh, you revert it and move on. If a hundred commits are sitting on top of it, reverting isn't an option, and you're untangling instead, which costs more than the original work and leaves you nothing to show for it.
Review cycles and inside knowledge
Fixing your pipeline doesn't make this smooth, and I'd rather say so than let the argument suggest otherwise.
Every merged change gets reviewed by the client. A lot of those comments come down to inside knowledge of the project, the history and context that lives in people's heads and in no document an agent can read. The changes that follow are often cosmetic. It's still a full cycle, and no model gets you out of it. Removing the biggest limit doesn't leave you with none. It shows you the next one.
What this means for planning
Measure the wrong thing, and you'll fund the wrong project. Before committing to an AI-assisted modernization program, I'd want three numbers that have nothing to do with your choice of model: how often your pipeline flakes, how long a run takes on the clock, and how many changes a week your reviewers can really absorb. Those three set your ceiling. The model decides how fast you hit it, and hitting your ceiling sooner isn't the same as raising it.
Be careful what a pilot tells you, too. Twenty pull requests won't show you any of this. At that size, the retries get absorbed without anyone noticing, and the whole thing looks like a win. A hundred thousand will show you. The problem isn't that the pilot lies. It's that the pilot works at a size where the real limit stays out of sight.
I don't think any of this is bad news. It means the thing holding you back is the thing you already control. Nobody gets to decide how good the models are next quarter. Everybody gets to decide what their test infrastructure looks like, and on a project like this one, that's quietly become the better investment of the two.
And that points at something bigger than CI. Look at where my time actually goes on this project. I scope the work, because the model can't hold the whole picture the way a person can. I write the plans and argue with them until I believe them. I keep the tests honest, because they're what lets an agent work without me watching it. None of that is writing code. All of it is preparing input.
That's the part of the job that grew rather than shrank. The better the input for the agent, the better the end result, and a flaky pipeline is just bad input arriving at the worst possible moment. It fails the same way a vague plan fails. It only fails later, after you've already paid for the work.
So when a client asks how much faster this makes things, the honest answer is that it goes as fast as you can merge. And merge speed isn't something that happens to you. It's input, like everything else here, and it's the piece most teams have never thought to prepare.
