How I Cut a 41-Minute CI Pipeline to 9 Minutes With Claude Code

TL;DR

Our CI pipeline took 41 minutes per push, so nobody pushed small commits anymore. I spent two afternoons with Claude Code profiling the pipeline instead of guessing at it, and got it down to 9 minutes. The wins weren't clever — they were boring things I'd been too impatient to measure.

The Problem

A 41-minute pipeline doesn't just cost 41 minutes. It changes how people work.

I noticed it in the shape of our commits. Instead of pushing a small fix and letting CI check it, people batched. Three days of work, one push, one giant PR. When that PR went red, you had no idea which of the fourteen changes broke it. So you'd push a fix, wait 41 minutes, guess again.

The math got ugly fast. Six engineers, roughly four pushes each per day, 41 minutes of wall clock per run. That's about 16 hours of pipeline time a day on a runner pool that could handle maybe 6 concurrent jobs. We were queuing. Some afternoons, "CI is slow" meant an hour of queue time on top of the 41 minutes.

I'd tried to fix this before, twice. Both times I did the same thing: opened the CI config, looked for something obviously dumb, added a cache key, declared victory. Both times it got 3-4 minutes faster and drifted back within a month.

The reason those attempts failed is embarrassing in hindsight. I never actually measured where the 41 minutes went. I looked at the YAML file and pattern-matched on things that "look slow." That's not profiling, that's vibes.

How I Solved It

The thing that finally worked was treating the pipeline like a performance bug in an application: get real timing data first, and don't touch a line of config until the data says where the time is.

Step 1: Get the data out of CI and into a file

Every CI provider exposes per-step timings through its API. I asked Claude Code to pull the last 50 runs on main and flatten them into something I could sort.

The important part of the prompt wasn't the task, it was the constraint:

Pull the last 50 successful pipeline runs from the CI API. For each run, extract every job and step with its duration in seconds. Write it to ci-timings.json. Do not analyze it yet, do not suggest fixes, and do not open the CI config. I only want the data.

That "do not suggest fixes yet" line matters more than it looks. If you ask an agent to fetch data and fix a problem in the same breath, it will start proposing fixes from the first thing it sees, and then everything downstream is an argument for that first guess. Separating collection from analysis is the same discipline you'd use profiling a slow endpoint.

The resulting script was maybe 60 lines:

import json, os, urllib.request

API = "https://ci.example.internal/api/v4"
TOKEN = os.environ["CI_TOKEN"]

def get(path):
    req = urllib.request.Request(f"{API}{path}", headers={"PRIVATE-TOKEN": TOKEN})
    with urllib.request.urlopen(req) as r:
        return json.load(r)

runs = [r for r in get("/pipelines?ref=main&per_page=50") if r["status"] == "success"]

rows = []
for run in runs:
    for job in get(f"/pipelines/{run['id']}/jobs"):
        rows.append({
            "run": run["id"],
            "stage": job["stage"],
            "job": job["name"],
            "queued_s": job["queued_duration"] or 0,
            "duration_s": job["duration"] or 0,
        })

with open("ci-timings.json", "w") as f:
    json.dump(rows, f, indent=2)

Nothing smart here. That's the point — the value was in having the numbers, not in how I got them.

Step 2: Let the median, not the average, pick the target

Then a second pass, on the file only:

Read ci-timings.json. Group by job name. For each job, report median duration, p90, and median queue time. Sort by median duration descending. Tell me what fraction of total wall-clock the top 3 jobs account for. No recommendations yet.

The output reordered my entire mental model:

Two things I had wrong:

1. I was sure build:docker was the villain. It's the one people complain about, because it's the one whose logs scroll for ages. It was second, and it was the most consistent job on the board.
2. lint had a median queue time longer than its runtime. It waited nearly 4 minutes to spend 2m50s. Nobody had ever mentioned this, because from the outside it just looks like "CI is slow."

Step 3: Fix in the order the data says

Integration tests: 18m40s → 4m10s. The suite ran serially in one job against one Postgres container. Every test file did a full schema teardown and rebuild — about 9 seconds of setup, times 74 files, which is 11 minutes of the 18 spent creating and dropping tables.

Two changes. First, shard across 6 parallel jobs by test file. Second, and this was the bigger one, replace teardown-and-rebuild with a per-test transaction that rolls back:

# Before: every test file rebuilt the schema from scratch (~9s each)
@pytest.fixture(scope="module", autouse=True)
def db():
    drop_all(); create_all(); seed()
    yield
    drop_all()

# After: schema built once per session, each test rolls back
@pytest.fixture(scope="session", autouse=True)
def schema():
    drop_all(); create_all(); seed()
    yield
    drop_all()

@pytest.fixture(autouse=True)
def isolated(db_connection):
    tx = db_connection.begin()
    yield
    tx.rollback()

I want to be honest about the agent's role here. Claude Code found the pattern — it read all 74 test files and reported that 71 of them used the module-scoped rebuild and only 3 needed real committed state (they tested transaction behavior itself, so rollback isolation would have been wrong). Finding those 3 exceptions by hand is exactly the kind of tedious full-directory read I would have skipped, and skipping it would have produced 3 mystery failures I'd have blamed on sharding.

Docker build: 9m55s → 2m30s. The Dockerfile copied the whole source tree before installing dependencies, so every single commit invalidated the dependency layer. Reordering it is CI 101 and I'd known about it for a year:

# Before: any source change busts the dependency cache
COPY . /app
RUN pip install -r requirements.txt

# After: dependencies cached until requirements.txt changes
COPY requirements.txt /app/
RUN pip install -r requirements.txt
COPY . /app

That's a four-line diff worth seven minutes a run. It sat there for a year because it wasn't anybody's job and it never looked urgent.

Lint queue: 3m55s → 0m20s. This one was pure scheduling, not code. lint, test:unit, and test:integration were all in the same stage competing for the same 6 runners, and integration was hogging them. Moving lint and type-checking into an earlier, cheaper stage meant they got runners immediately — and they now fail fast, killing the run in under 3 minutes when someone pushes a syntax error, instead of after 40.

Here's the shape of the change:

graph LR
  subgraph Before["Before — 41 min"]
    A1[lint] --> B1[deploy]
    A2[test:unit] --> B1
    A3[test:integration] --> B1
    A4[build:docker] --> B1
  end
  subgraph After["After — 9 min"]
    C1[lint + types<br/>fast stage] --> C2[test:unit + 6x integration shards]
    C2 --> C3[build:docker<br/>cached deps]
    C3 --> C4[deploy]
  end

Final numbers, measured the same way I measured the original — 50 runs, median, not one lucky green build: 41m10s → 9m05s. The fast-fail path is 2m40s.

Lessons Learned

1. The loudest job is rarely the slowest job. build:docker got all the complaints because its logs are noisy and it scrolls forever. It was never the top cost. Perceived slowness tracks how much output a step produces, not how much time it takes. Only timing data breaks that illusion.

2. Separate "measure" from "fix" in your prompts. This is the single habit that changed my results with Claude Code. When one prompt says gather data and propose a fix, the model anchors on the first plausible cause and everything after is post-hoc justification. Two prompts with a file in between gives you an artifact you can check independently. Same reason you don't let a profiler also write the patch.

3. Queue time is invisible and it's real. Nobody logs "waited 4 minutes for a runner." It doesn't show up in the job duration your dashboard displays. On our pipeline, queue time was 11 of the 41 minutes — a quarter of the problem, and 100% of it was solvable by reordering stages rather than making anything faster.

4. Agents are worth the most on the tedious-but-exhaustive parts. The clever part of this fix (transactions instead of rebuilds) took me 20 seconds to think of. The valuable part was reading all 74 test files and correctly identifying the 3 that couldn't use it. I'd have sampled five files, assumed the rest matched, and shipped a broken sharding config. Use the agent where completeness matters and attention runs out.

5. If you didn't measure it the same way twice, you didn't fix it. My two previous attempts "worked" — I ran the pipeline once, it was faster, I moved on. Both regressed within a month because the improvement was inside the normal variance. Median across 50 runs, before and after, or you're just telling yourself a story.

What's Next

Two things I haven't done yet.

The first is a regression guard: a weekly job that reruns the same timing collection and opens an issue if the median creeps past 12 minutes. Every pipeline optimization I've ever seen decays, because nothing watches it. A number nobody looks at goes bad quietly.

The second is the integration suite itself. Sharding hid the real problem — the suite is still 18 minutes of work, I just bought 6 machines to do it. A decent chunk of those tests are integration tests only because they were easier to write that way, not because they need a database. That's a bigger project and honestly a less satisfying one, which is probably why it's still on the list.

Wrap-up

If your pipeline is slow and you've "looked at the config" without pulling per-step timings, you're where I was for a year. Spend 30 minutes getting real numbers into a file first. In my case the biggest fix was four lines of Dockerfile reordering I already knew about, and it stayed unfixed until the data made it embarrassing.

If you try this on your own pipeline, I'd genuinely like to hear what your top-3 jobs turned out to be — my bet is at least one of them surprises you. Drop it in the comments. 🚀

Follow me here on Dev.to if you want more posts like this — I write up what actually worked (and what didn't) when I hand real engineering work to AI coding agents.

Versions used: Claude Code v2.x, Python 3.13, pytest 8.x, Docker 27.x.