Your pipeline failed but your YAML was fine: how a Gitaly timeout broke GitLab CI on Friday

Friday afternoon. You push a commit, the pipeline kicks off, and instead of green checks you get this:

"CI configuration fetch from Gitaly timed out. This may indicate Gitaly service slowness or an outage."

Your first instinct is to open your .gitlab-ci.yml and hunt for the typo. Stop. There is no typo. Your config was never read at all.

What actually happened

On October 16, GitLab.com had a rough afternoon. At 11:33 AM UTC, GitLab posted that CI/CD was failing for some customers with exactly that error, and that the UI was slow too. They applied a temporary mitigation at 12:28 PM UTC and marked it resolved at 1:21 PM UTC. One hour and forty-eight minutes.

Then, later the same day, a second incident: 503s across the website, the API, Git operations, both registries, Pages, and the hosted runners on Linux, Windows, and macOS, from 4:10 PM UTC to 5:24 PM UTC.

Two incidents, one afternoon. The first one is the interesting one, because the error message tells you exactly where the system is coupled.

Why creating a pipeline reads your repo

Here is the part most people never think about. Your .gitlab-ci.yml lives in your repository. When you push and a pipeline is created, GitLab has to go read your config out of the repo, merge it with any includes, and only then can it build the list of jobs. There is no warm copy sitting next to the scheduler. Every pipeline starts with a read.

All Git reads in GitLab go through a service called Gitaly. Think of it as the front desk for every repository: instead of each part of GitLab opening raw Git data directly, they call Gitaly over RPC and Gitaly does the reading.

So the chain for every single pipeline looks like this:

push -> pipeline created -> fetch config from the repo via Gitaly -> parse and merge -> schedule jobs

That fetch step is synchronous. It sits on the critical path, and it has a timeout. When Gitaly got slow on Friday, the fetch hit the timeout, and pipeline creation failed before a single job was scheduled. Nothing ran. Nothing could run. The config was fine; the reader was sick.

That is also why the UI was slow at the same time. The same Gitaly layer serves the reads behind project pages, file views, and merge requests. One slow layer, many symptoms.

How to read this error

This is the practical skill the incident teaches. Not every red pipeline means the same thing, and the fix depends on which kind of red you got.

If your YAML has a syntax error, GitLab tells you your YAML has a syntax error. It names the file and the line. That is a config problem, and the fix is in your editor.

"CI configuration fetch from Gitaly timed out" is a different sentence. It says: I could not fetch. The failure is in the reading, not the writing. Debugging your YAML against this error is like proofreading a letter the post office lost.

So the first move is not your editor. It is the status page. If GitLab is having a Gitaly moment, no amount of YAML polishing will turn that pipeline green.

What to do while you wait

You still want that pipeline to run. The instinct is to hammer the retry button, or re-push the same commit five times. That instinct makes the problem worse.

Every retry is another config fetch through the same struggling layer. Thousands of developers retrying at once is a thundering herd aimed directly at the thing that is already down. (I wrote about why retries need jitter a couple of weeks ago: https://dev.to/kashif_manzer/why-your-retries-need-jitter-the-thundering-herd-explained-3eje. This is exactly the scenario it is for.)

The calmer version: retry with backoff, and let the error message decide whether retrying even makes sense. Here is a small wrapper I keep around for triggering pipelines from scripts:

import random
import time

import requests

API = "https://gitlab.com/api/v4"
HEADERS = {"PRIVATE-TOKEN": "your-token-here"}

def create_pipeline(project_id, ref="main", tries=5):
    delay = 2  # seconds
    for attempt in range(1, tries + 1):
        r = requests.post(
            f"{API}/projects/{project_id}/pipeline",
            headers=HEADERS,
            json={"ref": ref},
            timeout=30,
        )
        if r.ok:
            return r.json()
        # Only retry fetch-side timeouts. Anything else fails fast.
        if "gitaly" not in r.text.lower() or attempt == tries:
            r.raise_for_status()
        wait = delay + random.uniform(0, delay)
        print(f"attempt {attempt}: config fetch timed out, backing off {wait:.1f}s")
        time.sleep(wait)
        delay *= 2

The logic is simple. If the failure mentions the config fetch timing out, wait and try again with a growing, jittered delay. If it fails for any other reason, fail fast, because retrying a genuinely broken config is just burning API calls.

Two more habits that pay off during incidents like this:

1. Check the status page before you touch your code. Five seconds on the GitLab status page saves twenty minutes of phantom debugging.
2. Keep your includes lean. Every include is another file GitLab has to fetch and merge before your pipeline exists. A config that pulls in six included files gives the fetch step six files to read before a single job is scheduled.

The design lesson

Zoom out and the incident is a textbook coupling story. A config file feels static, like something the system should just know. But GitLab treats it as dynamic: fetched fresh on every pipeline, through a network service, on the critical path, with a timeout.

Dynamic fetching buys you real things. Edit the YAML, push, and the next pipeline picks it up with no cache to invalidate and no stale config to chase. The price is a dependency: your pipeline cannot exist faster than the slowest read in its setup path.

Every system has a version of this. A service that reads its feature flags from a remote store on every request. A deploy that pulls its manifest from an API before it starts. The question is always the same: is this read on the critical path, and what happens when it is slow instead of down? Slow is the cruel one. Down fails fast. Slow burns through timeouts, holds connections open, and takes the UI down with it.

Friday's error message was honest about all of this. It named the layer, named the timeout, and told you it might be slowness rather than an outage. Most error messages are not that generous. When one is, believe it.

What is the fetch-on-the-critical-path in your own system, the read everything quietly depends on? Have you ever watched it get slow?