1,400 PRs and 800k Lines Deleted: How OpenAI Cleans Up Its Code with Agents (and What You Should Steal)

This is a retelling with commentary. The original is Rodrigo Roim's post on the OpenAI Developers blog, October 14, 2026: Maintaining software with the GPT-6 family. Everything under "My take" is my own analysis, not the author's position.

TL;DR

• Five engineers on OpenAI's Core Experimentation team, working part-time over two weeks, merged 1,400+ PRs and deleted 800k+ lines of code that GPT-6 Astra found and prepared for removal via Codex.
• In a separate pass, the agent found hundreds of stale experiments and feature flags, including one "permanent" flag that was burning ~0.42% CPU in the main service.
• The cheaper GPT-6.1 Sol at Ultra effort produced 92% of Astra's findings at 14% of the cost.
• CI got roughly twice as fast: median from ~10 to 4.5 minutes, p90 from 12.5 to 6 minutes.
• For context: Anthropic's recent Claude Opus 5.5 makes the same bet on long, codebase-wide tasks and tells the same cost story ("beats the competitor's flagship at a fifth of the cost"). I break down what can and can't be compared.
• The main message of the post isn't "agents will do everything" — it's "start small, review, scale up, and feed review feedback into the next prompt."

Context: agents sped up development — and tech debt too

Here's a paradox people rarely talk about: when code gets written faster, junk accumulates faster too. Unused functions, long-finished A/B tests, flaky tests. Roim's team builds OpenAI's internal experimentation platform — everything from frontend to model-serving optimizations — so they have a lot of flags and experiments.

The post covers three maintenance scenarios and a cost comparison between models.

1. Removing dead code

The old way. The classic approach is heavy static analysis plus runtime logging. The author admits that even with that and careful reviews, there were incidents caused by mistakenly deleted code. Reflection breaks static analysis, and a "dead" branch might turn out to be an emergency switch for some future incident.

The intermediate step (GPT-5.6 Sol). The model was given a specific area and clear validation criteria — and it wrote the static analysis and removed the code itself. These targeted prompts turned out to be easier to maintain than the homegrown analysis system, with comparable reliability.

With GPT-6 Astra, the team tested whether the model could handle a vague request. The prompt was literally a sentence or two (typos included): "find and clean up dead code; feel free to check Slack, Statsig, or any other source to confirm it's really unused."

What the model did:

• dug through PR history;
• pulled context from Slack threads and internal trackers;
• checked production profiling tables to confirm the deleted lines were never executed;
• attached a rationale to every deletion — which sped up reviews dramatically.

Result: 1,400+ PRs, 800k+ lines deleted. Reviews were distributed to the teams owning the relevant code. Tangible wins: one service's container image shrank from 3.31 GB to 1.72 GB (−48%), and about 400 visual snapshot variants disappeared from the test suite.

2. Stale experiments — the harder problem

A stale experiment isn't the same as dead code. It may still run in production, and its eligibility conditions could theoretically change at any moment. Static analysis is nearly useless here. The team previously used deterministic scripts with database queries and knew they were missing many genuinely stale experiments (false negatives).

The author's approach is interesting in itself: he didn't know where to start, so he asked the model to write the prompt for him. The result was a short instruction along these lines:

• find substantial dead code tied to constant or abandoned Statsig gates/configs in two specific projects;
• check against the live configuration, git history, and overlapping PRs;
• preserve production behavior, safeguards, and killswitch flags;
• ignore SDK fallbacks and local overrides;
• one small draft PR per cleanup, with a link to the flag's console page.

Then came the loop: the author comments on the PRs → asks the model to incorporate the comments → asks it to write a prompt to scan the entire repository, plus help with prioritization (he was worried about drowning in findings).

The final "big" prompt (using the /goal command) required the agent to:

• audit all Statsig projects and inventories, not just the ones labeled stale;
• look for indirect usages: enums, constants, wrappers, aliases, schemas, dynamically assembled names, consumers in other services;
• factor in the CPU cost of each check based on production telemetry;
• produce a markdown report split into parallel workstreams by owner and file, with evidence and costs;
• not touch live configurations or services.

Result: the top of the report was dominated by big efficiency wins — experiments nobody had considered stale for years. After manual verification, hundreds of experiments and flags were removed.

The most telling story

Years ago, someone created a flag that let internal developers override all other new-feature flags — for testing. It was implemented as a single line of Python mapping that flag to every feature, buried among hundreds of similar lines. Inconspicuous.

Astra figured out that this line caused a serious CPU fan-out, estimated the cost — but didn't delete it on its own. It escalated to a human reviewer: the flag was marked "Permanent" in the console and could have had an operational purpose. After review, the flag was removed: estimated savings of ~0.42% CPU in the main affected service.

3. Cost: GPT-6 Astra vs. GPT-6.1 Sol

The author ran the same stale-experiment prompt on subdirectories of one repository — 6,000+ files, ~1M lines of Python, TypeScript, Go, and Rust — and manually validated the findings.

The cost includes research subagents and automatic follow-up turns by Codex.

The main discrepancy between the models at Ultra: Sol missed constant flags used as dependencies of other flags, while Astra caught them. The author's takeaway: such error patterns can be compensated with prompting to get close to the expensive model's results with the cheap one. But new model releases can make those "crutches" obsolete, so prompts need revisiting.

4. Three more ideas from the post

The author suggests starting these on GPT-6.1 Sol Medium and raising effort if results look promising.

1. Flaky tests. Scan recent CI runs, find unstable tests, try a fix for each, and open a PR if it works. Key caveat in the prompt: the goal is obvious fixes for simple problems, not exhaustively fixing everything.
2. Cleaning up AGENTS.md. Find an instruction in recent sessions that misled the agent, then A/B test it: replay the original prompt on the same repo snapshot with and without that instruction, comparing quality, tool calls, tokens, runtime, and cost. Essentially an eval for your own context file.
3. Reducing CI load. Ask for a table of ideas to cut CI load, ranked by estimated CI minutes saved per day based on run history.

The third idea led to about 30 optimization PRs: median CI from ~10 to 4.5 minutes, p90 from 12.5 to 6. One standalone fix — adding a missing mock — cut a 63-test suite from 592.6 to 29.8 seconds (−95%, all tests kept).

For comparison: what Anthropic claims about Claude Opus 5.5

To avoid seeing the topic through a single vendor's eyes, it's worth putting a competitor's recent release side by side. On September 22, 2026 — about three weeks before the OpenAI post — Anthropic released Claude Opus 5.5. The positioning is very similar: long agentic tasks across the whole codebase, migrations, audits, reviews — with an emphasis on cost.

What in the announcement relates to code maintenance

• Auditing and fixing a 200k-line codebase took one early tester under three hours. The previous Opus 5 needed more than 20 hours and 2.5× the tokens for the same job.
• A 680k-line migration — in under a day. Anthropic estimates it would have taken an engineering team weeks.
• Optimization without behavior changes. Asked to cut load times across every page of a web app, Opus 5.5 succeeded 39 times out of 40, while Opus 5 made smaller improvements that also altered the app's behavior. For maintenance work, this is arguably the most important metric in the entire announcement.
• Rewriting HAProxy from C to Rust in an internal test: both models (Opus 5.5 and the pricier Fable 5.1) passed nearly all of the project's regression tests, but Opus 5.5 finished in 9.5 hours versus 12 and cost 51% less.
• Code review: according to Deloitte, even at its lowest effort the model caught 72% of known bugs versus 56% for Opus 5 at high effort — with fewer false alarms.
• Orchestration: a Stripe engineer describes a multi-day rebase of 40 stacked PRs where one session directed a dozen others, and all 40 PRs passed CI the next day.

Numbers that can be lined up against the OpenAI post

According to Anthropic, at default effort (medium) Opus 5.5 beats Astra's top FrontierCode score at roughly a fifth of the cost per task, and matches Astra on Terminal-Bench at about 40% of the cost. Astra's figures in the table are those published by OpenAI, not Anthropic's own measurements.

On safety, Anthropic says the model is much less likely to take hard-to-reverse actions or step outside its given boundaries, and in a new containment test it attempted to circumvent boundaries about 85% less often than Opus 5. The company is also candid that the model often suspects it's being evaluated, which limits confidence in how it will behave in real-world settings.

What this comparison tells us

1. Both companies tell the same cost story. OpenAI: "GPT-6.1 Sol delivers 92% of Astra's results at 14% of the price." Anthropic: "at medium effort we beat the competitor's flagship at a fifth of the cost" — and Factory, quoted in the announcement, says this is the first model they'd default to at medium. The race has shifted from peak scores to cost per useful result. That's good news for us: "cheap mode by default, expensive mode surgically" is now advice from both vendors.

2. A direct comparison isn't valid — and that matters. OpenAI has an internal case study with real findings validated by hand. Anthropic has benchmarks and quotes from early testers the vendor picked itself. Neither is an independent evaluation. Competitor numbers in the tables are taken "as reported by the other side," with different harnesses and effort settings. Anthropic itself admits in the announcement that at this capability level, benchmark margins increasingly fail to reflect real-world differences.

3. Read the footnotes. A small but telling example: the announcement's table lists 54.4% for Opus 5.5 on FrontierCode (the max-effort result), while the text below says 54.6% at medium. So medium comes out slightly higher than max. That may well be noise or different runs, but these are exactly the kinds of discrepancies worth noticing before drawing conclusions.

4. Anthropic names the metrics missing from the OpenAI post. "Improved without changing behavior" (39 of 40) and "fewer false alarms in review" are exactly what I argue below: for maintenance work, false-positive rate and behavior preservation matter more than the number of PRs opened. It's a shame neither vendor publishes these metrics systematically.

How to compare models on your own code

Since vendor numbers can't be taken at face value, the only honest benchmark is your repository. A mini-protocol you can run in a day:

1. Pin a snapshot of the repo (a commit) and one prompt — e.g., finding stale flags in a single directory.
2. Run it on 2–4 configurations: different models and effort levels.
3. For each run, count accepted findings, rejected findings (false positives), review time, and API cost.
4. Analyze the discrepancies, as the OpenAI author did: which classes of findings does the cheap configuration miss — and can a prompt close the gap?
5. Save the snapshot, prompt, and list of reference findings — that's your eval for the next model release.

The key number isn't "cost per finding" but cost per accepted finding, including reviewer time.

My take: what's impressive in these numbers, and what to read with caution

The numbers check out

I recalculated: 131/142 ≈ 92%, $15.12/$109.06 ≈ 13.9% — the claimed "92% of findings at 14% of the cost" is honest. Median CI −55%, p90 −52%, image −48%. No arithmetic sleight of hand.

But it's a vendor case study — and that matters

• The post is written by the company that sells these models. That doesn't make it false, but the sample is successful scenarios inside OpenAI, in an ideal environment for an agent: telemetry, profilers, access to Slack and the flag console, unlimited model access.
• No false-positive rate. We know how many findings were accepted, but not how many were rejected or how much time it took to triage them. "Cost per finding" in the table only counts API spend, not reviewer hours. For 1,400 PRs, human time is almost certainly worth more than $109.
• A single validator. In the model comparison, findings were checked by the author himself. That's fine for a blog post, but it's not a benchmark.
• Lines of code are a vanity metric. 800k lines / 1,400 PRs ≈ 570 lines per PR on average. Impressive, but the real value shows in other numbers: image size, CPU, CI time. Good on the author for including them.
• The author admits that some proposals added more complexity than they were worth — prioritization and measurement stayed with humans.

Why deleting code is the ideal task for an agent

I don't think it's a coincidence that the loudest agent success story is not a new feature but deletion. The task has a rare combination of properties:

• Verifying is easier than doing. Finding dead code in a million lines is slow and boring. Confirming that a specific deletion is safe, when evidence is attached (zero calls in the profiler, flag at 100% for two years, a thread saying "experiment closed"), takes a couple of minutes. Agents are strongest where verification is cheaper than generation.
• Bounded damage. A deletion goes through tests and review and is easy to roll back with git revert. Compare that with a new feature, where the mistake may be in the idea itself.
• Measurable outcome. Megabytes of image, percent of CPU, minutes of CI. You can calculate whether it paid off.
• It's work people put off for years. Nobody finds it interesting and it doesn't get anyone promoted. That's why the biggest "free" wins pile up here.

The practical takeaway: if you're looking for where to first deploy an agent on your team, code maintenance is a better candidate than generating new code.

The bottleneck shifts from finding to reviewing

Rough math. Say a PR review takes 10 minutes on average (realistic with evidence attached; longer without). 1,400 PRs × 10 minutes ≈ 230 person-hours. That's more than a month of one engineer's work. The API cost is a rounding error by comparison.

That changes the whole economics:

• The scarce resource is reviewer attention, not tokens. Optimize the PR format: short diff, clear rationale, links to evidence, nothing extra.
• Without a code ownership map, this doesn't scale. OpenAI routed reviews to owning teams. If you don't have an up-to-date CODEOWNERS or equivalent, 1,400 PRs will sit in a queue.
• A prioritized report matters more than PRs. The author was right to ask the agent for a report with estimated benefits first. An agent that opens 500 unranked PRs shifts work onto people rather than removing it.

Marginal cost per finding: the most interesting part of the table

The post compares average cost per finding. But for deciding "which model to use," the marginal cost matters more: how much each additional finding costs when moving to a more expensive configuration.

The marginal cost per finding jumps roughly 40×. A classic diminishing-returns curve.

But there's a nuance that's easy to miss. Those 11 findings only Astra caught are constant flags acting as dependencies of other flags. These are the most tangled cases, and possibly the most valuable. So the right question isn't "which model is cheaper" but "what does a finding the cheap model misses cost me?" For tech debt, usually not much — so Sol Ultra looks like a sensible default. For a security audit, the answer might be different.

The working strategy that follows: a bulk pass with the cheap model, and the expensive one applied surgically to the known blind spots you discovered by comparing results.

An agent doesn't replace static analysis — it complements it

The agentic approach is powerful because it can connect unstructured sources: Slack, PR history, trackers, telemetry. A deterministic analyzer will never do that. But the agent has a flip side: non-determinism. Two runs of the same prompt can produce different lists of findings. That's fine for a one-off cleanup, not for guarantees.

I think the most viable setup is hybrid:

1. The agent discovers a class of problems and proves specific instances.
2. The discovered pattern is turned into a deterministic check: a lint rule, a CI job, a script. The agent can write it too — as happened with GPT-5.6 Sol in the post.
3. The check keeps the problem from coming back, while the agent goes hunting for new classes.

That way, knowledge gained through an expensive non-deterministic process gets locked into a cheap deterministic one.

Prompts are code — and they rot too

One offhand remark by the author is, in my view, underrated: prompt tweaks that compensate for a model's weaknesses become obsolete with the next release. That means:

• prompts for recurring tasks should be stored in the repository and versioned, like code;
• they need evals: a fixed repo snapshot plus a set of known correct findings. Otherwise, when you switch models, you won't know whether things got better or worse;
• the post's idea of A/B testing AGENTS.md instructions is exactly such an eval in miniature. It deserves to become a regular practice, not a one-off experiment.

Without telemetry, the agent is guessing

Notice what Astra relied on for its strongest findings: production profiles, CPU per flag check, CI run history. Without that data, the agent could only say "looks unused" — the same thing static analysis says, just more expensively.

A non-obvious consequence: investing in observability becomes a prerequisite for agentic code maintenance. If you don't have production profiling and per-flag metrics, start there. Otherwise you'll have an agent working blindfolded.

Chesterton's fence, agent edition

There's an old principle: don't take down a fence until you understand why it was put up. The "Permanent" flag story is a textbook example: the agent found an expensive line, calculated its cost, but stopped when it saw signs of intent and handed the decision to a human.

That behavior wasn't accidental: the prompt set it up ("preserve killswitches and safeguards"). The practical takeaway: escalation criteria should be spelled out as carefully as deletion criteria. The agent needs to know not only what to look for, but which signals mean stop: "permanent" or "do not remove" labels, mentions in postmortems, links to incidents, code in disaster-recovery paths.

And one more thing the post doesn't mention: deleted code and closed experiments are part of institutional memory. A dead branch is sometimes the only place showing how the system used to work. The answer isn't to avoid deletion but to write good PR descriptions. An agent that writes a rationale for every deletion is also building an archive where, a year later, you can find out "where did X go."

A loop worth thinking about

The post opens by admitting that agents sped up development, which increased the maintenance load. And it ends with agents doing the maintenance too. That's a loop: agents produce code faster than humans can absorb it, and agents clean that code up.

That isn't necessarily bad, but it changes the engineer's role: less writing, more setting constraints, checking evidence, and making decisions. The good news: those skills always separated seniors from juniors. The bad news: juniors will have a harder time gaining experience on the routine work agents now take over. How teams will grow people in this environment is a question the industry doesn't have an answer to yet.

What actually transfers (even if you don't have GPT-6)

The most valuable part of the post isn't the models but the patterns for working with an agent. They apply to any coding agent:

1. Demand evidence, not just a diff. Every deletion comes with a rationale and links (PR, thread, metric). That's what makes 1,400 PRs reviewable.
2. Explicitly list what must not be touched. Killswitches, emergency branches, "Permanent" flags. The 0.42% CPU flag story is the perfect example: the agent found the problem but escalated instead of deleting.
3. Small draft PRs, not one mega-commit. Easier to review, easier to roll back, easier to route to owners.
4. Meta-prompting. Asking the model to write the prompt for the next pass, with review feedback baked in, is a cheap and effective way to iteratively improve instructions.
5. Start with the cheap model on a small scope. Compare configurations on one task — Medium may well be enough.
6. Report → prioritization → parallel workstreams. An agent that writes a report with estimated benefits first is more useful than one that immediately opens 500 PRs.

On safety — what the post says too little about

"Don't touch live configs" in a prompt is a wish, not a guarantee. If the agent has write access to the flag system, sooner or later it will use it in a way you didn't expect. Practical conclusions:

• give the agent read-only access to Slack, trackers, the flag console, and telemetry;
• keep merging and config changes in the hands of humans and CI;
• remember that Slack threads and tickets are untrusted input: an agent that reads everything is vulnerable to instructions hidden in that text.

How not to let your flags get to this point

An agent cleaning up flags that piled up for years is the cure. Prevention is cheaper:

• every flag has an owner and an expiration date;
• experiment flags and operational killswitches are different types, with different lifecycle rules;
• a recurring (even monthly) agent task: "find flags that have been at 100% or 0% for 30+ days and prepare a report" — exactly what the post describes, but in small, regular doses.

Wrap-up

OpenAI's post isn't about "agents will replace engineers" — it's about code maintenance turning into a loop: the agent finds and proves → a human decides → feedback improves the next prompt. The strongest point is about cost: a 7× price difference between models for 92% of the result means that for routine tech debt, an expensive flagship may simply be overkill. The Claude Opus 5.5 announcement points the same way — and that convergence is more telling than any single vendor's benchmarks.

Want to try it? Take one flaky test or one stale flag and ask an agent to find it, prove it, and open a small draft PR. If it works, expand to a directory. Then to the repository.

Are you already letting agents clean up tech debt? What's your PR acceptance rate? Share in the comments 👇

Sources:

• Maintaining software with the GPT-6 family, OpenAI Developers Blog, Oct 14, 2026
• Introducing Claude Opus 5.5, Anthropic, Sep 22, 2026