Four Stages, Zero needs: Reconstruct the GitLab DAG Before You Merge

You opened the merge request expecting a cleanup. Instead .gitlab-ci.yml lost every needs: key. Four stages remain. Every job in test now waits for every job in lint, including the one that only prints a license header.

GitLab did not get slower. The graph did. Sequential stages are a barrier, not a design. This walkthrough restores a directed acyclic graph from the YAML you still have, then writes needs: you can defend in review.

The cost of a “simpler” file

A stage list is an ordering constraint. Job A in lint and job B in test do not share a filesystem. They share a clock. If B does not consume A’s artifacts, that clock is fiction.

You feel it in the merge train. Lint-format finishes in forty seconds. Unit tests still sit idle because a container scan in the same earlier stage is pulling an image. Nobody asked for that wait. The rewrite just deleted the edges that used to skip it.

Do not start by asking a model to “make CI faster.” Start by naming what each job reads.

What GitLab actually waits on

Two mechanisms exist. Learn both before you type YAML again.

1. Stages. A job in stage n waits until every job in stage n-1 finishes, unless needs: says otherwise.
2. needs:. A job may start when the named jobs finish, even if other jobs in earlier stages are still running.

resource_group, interruptible, and parallel:matrix are different knobs. They do not replace a missing edge. They also do not create one. If the assistant flattened the file “so stages are obvious,” it traded a DAG for a queue.

Confirm empty needs: [] against your GitLab CI YAML reference. Behavior is version-specific enough that you should not cargo-cult a blog snippet. The reconstruction below assumes you will lint the result on your project, not on a gist.

Step 1 — Dump the jobs you still have

Work from the commit in the merge request. Do not work from a chat paste.

git rev-parse HEAD
git show HEAD:.gitlab-ci.yml | wc -l

If the live file is only includes, stop. This method parses one document. Catalogs that live in another project need a different checkout. For a single file, extract job-shaped keys.

The script below is a proposal. It needs PyYAML. It does not expand include:, does not evaluate rules:, and does not talk to the GitLab API.

#!/usr/bin/env python3
"""dag_from_ci.py — proposed static graph extract for one .gitlab-ci.yml."""
from __future__ import annotations

import sys
from collections import defaultdict
from pathlib import Path

import yaml

RESERVED = {
    "default",
    "include",
    "stages",
    "variables",
    "workflow",
    "image",
    "cache",
    "before_script",
    "after_script",
    "pages",  # keep if you treat pages as a real job; delete this line if so
    "spec",
}

JOB_MARKERS = {"script", "trigger", "needs", "extends", "stage", "parallel"}


def load(path: Path) -> dict:
    data = yaml.safe_load(path.read_text()) or {}
    if not isinstance(data, dict):
        raise SystemExit("root YAML value is not a mapping")
    return data


def is_job(name: str, body) -> bool:
    if not isinstance(body, dict):
        return False
    if name.startswith("."):
        return False
    if name in RESERVED:
        return False
    return bool(JOB_MARKERS.intersection(body))


def jobs_by_stage(data: dict) -> dict[str, list[str]]:
    stages = data.get("stages") or ["test"]
    grouped = {s: [] for s in stages}
    unknown = []
    for name, body in data.items():
        if not is_job(name, body):
            continue
        stage = body.get("stage", stages[0] if stages else "test")
        if stage not in grouped:
            unknown.append((name, stage))
            continue
        grouped[stage].append(name)
    if unknown:
        print("# jobs with stage not in stages:", file=sys.stderr)
        for name, stage in unknown:
            print(f"#   {name}: {stage}", file=sys.stderr)
    return grouped


def explicit_needs(body: dict) -> list[str]:
    raw = body.get("needs") or []
    names = []
    for item in raw:
        if isinstance(item, str):
            names.append(item)
        elif isinstance(item, dict) and "job" in item:
            names.append(item["job"])
    return names


def main() -> None:
    path = Path(sys.argv[1] if len(sys.argv) > 1 else ".gitlab-ci.yml")
    data = load(path)
    grouped = jobs_by_stage(data)
    print("stages:")
    for stage, names in grouped.items():
        print(f"  {stage}: {', '.join(names) or '(empty)'}")
    print("explicit_needs:")
    for name, body in data.items():
        if is_job(name, body):
            print(f"  {name}: {explicit_needs(body) or '(none)'}")


if __name__ == "__main__":
    main()

Run it on the branch, not on main, if main still has the old graph.

python3 dag_from_ci.py .gitlab-ci.yml

If explicit_needs prints (none) for every job, you are looking at a queue. That is the bug. The stages may still be named after real work.

Step 2 — Draw the hidden barrier graph

When needs: is absent, GitLab still has edges. They are just implicit: every job in stage n depends on every job in stage n-1. Write that down. A mermaid file is easier to argue with than a feeling.

Add this function to the same proposed script, then print it.

def barrier_edges(grouped: dict[str, list[str]]) -> list[tuple[str, str]]:
    stages = list(grouped)
    edges = []
    for i in range(1, len(stages)):
        prev, cur = stages[i - 1], stages[i]
        for src in grouped[prev]:
            for dst in grouped[cur]:
                edges.append((src, dst))
    return edges


def emit_mermaid(grouped: dict[str, list[str]], edges: list[tuple[str, str]]) -> str:
    lines = ["```

mermaid", "flowchart LR"]
    for stage, names in grouped.items():
        lines.append(f"  subgraph {stage}")
        for n in names:
            lines.append(f"    {n}[{n}]")
        lines.append("  end")
    for src, dst in edges:
        lines.append(f"  {src} --> {dst}")
    lines.append("

```")
    return "\n".join(lines)

Count the edges. A four-stage file with three jobs per stage produces nine implicit waits between each pair of stages. That is twenty-seven arrows for work that might only need four. Paste the mermaid into the MR description. Reviewers can see the serialization. You do not have to narrate it.

Step 3 — Recover edges from artifacts, not from vibes

The useful graph is smaller. A downstream job should wait only on jobs whose outputs it reads.

Scan each script / before_script for paths you already declare under artifacts:paths. This is heuristic. Label it that way in the MR. It will miss S3 downloads and it will miss needs: used only for ordering.

import re

PATH_RE = re.compile(r"[\w./-]+\.(xml|json|txt|whl|tar\.gz|html|lcov)")


def script_lines(body: dict) -> str:
    chunks = []
    for key in ("before_script", "script", "after_script"):
        val = body.get(key) or []
        if isinstance(val, str):
            chunks.append(val)
        else:
            chunks.extend(str(x) for x in val)
    return "\n".join(chunks)


def artifact_paths(body: dict) -> set[str]:
    arts = body.get("artifacts") or {}
    paths = arts.get("paths") or []
    reports = arts.get("reports") or {}
    extra = []
    for v in reports.values():
        if isinstance(v, str):
            extra.append(v)
        elif isinstance(v, list):
            extra.extend(str(x) for x in v)
    return set(paths) | set(extra)


def proposed_needs(data: dict) -> dict[str, list[str]]:
    producers = []
    for name, body in data.items():
        if is_job(name, body):
            for p in artifact_paths(body):
                producers.append((name, p))
    result = {}
    for name, body in data.items():
        if not is_job(name, body):
            continue
        text = script_lines(body)
        found = set()
        for producer, path in producers:
            if producer == name:
                continue
            if path and path in text:
                found.add(producer)
        result[name] = sorted(found)
    return result

Print a three-column table in the terminal. Left: job. Middle: implicit barrier parents. Right: artifact-derived parents. The MR discussion is the diff between those two columns.

If the right column is empty, you do not have a graph yet. You have guesses. Either declare artifacts:paths properly or write needs: by hand from how the team actually ships.

A throwaway parser host is useful here. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and a free server option can draft candidate edges from script text and run dag_from_ci.py off your runners. Treat that box as a sketching desk. It does not expand GitLab includes, and it does not record a pipeline ID.

Step 4 — Emit needs: that GitLab will accept

Do not dump every recovered name into one job. Cross-stage needs: is allowed. Same-stage needs: is allowed when you want an explicit order inside a stage. Optional jobs are the trap.

If rules: can skip lint:docs, then test:unit must not hard-need lint:docs. Use optional: true on that edge, or drop the edge. A pipeline that never starts because a needed job was skipped is worse than a sequential one.

Proposed fragment. Replace names. Keep it smaller than the barrier graph.

# proposed reconstruction — lint against YOUR project
stages: [lint, test, build, deploy]

lint:code:
  stage: lint
  script: ["echo lint"]

lint:docs:
  stage: lint
  script: ["echo docs"]
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: always
    - when: never

test:unit:
  stage: test
  needs:
    - job: lint:code
      artifacts: false
    - job: lint:docs
      optional: true
      artifacts: false
  script: ["echo pytest"]

build:image:
  stage: build
  needs:
    - job: test:unit
      artifacts: true
  script: ["echo build"]
  artifacts:
    paths: ["dist/"]

deploy:stg:
  stage: deploy
  needs:
    - job: build:image
      artifacts: true
  script: ["echo deploy"]
  environment:
    name: staging
  resource_group: staging

artifacts: false on lint edges is deliberate. You wanted ordering or a signal, not a tarball. resource_group stays on deploy. The assistant often deletes that key while flattening stages. Put it back before anyone double-deploys staging.

Lint on the project after the edit. Local yamllint does not expand includes the way GitLab does.

# use the CI Lint in the GitLab UI or the project lint API
# do not treat a local YAML parse as a pipeline

Step 5 — Cycle check before you push

needs: can form a cycle. GitLab will reject it. Catch it in the sketch so the lint error is not your first signal.

def has_cycle(edges: list[tuple[str, str]]) -> bool:
    graph = defaultdict(list)
    indeg = defaultdict(int)
    nodes = set()
    for a, b in edges:
        graph[a].append(b)
        indeg[b] += 1
        nodes.add(a)
        nodes.add(b)
    q = [n for n in nodes if indeg[n] == 0]
    seen = 0
    while q:
        n = q.pop()
        seen += 1
        for m in graph[n]:
            indeg[m] -= 1
            if indeg[m] == 0:
                q.append(m)
    return seen != len(nodes)

If has_cycle is true, you over-connected. Typical mistake: both test:unit and test:integration need each other because both scripts mention coverage.xml. Pick one producer. Coverage files need a single writer.

Decision table for the MR

Fill this. If a cell is “unknown,” do not merge the flattened file.

The interesting column is “May run beside.” That is the parallelism you deleted. Restore only those edges. Leave the rest unconnected.

Limits

This reconstruction does not become GitLab. It is a static reading of one YAML document plus a human table.

• It will not expand include:, include:component, or include:project.
• It will not evaluate workflow:rules or job rules:.
• It will not model parent-child pipelines or trigger:.
• It will not prove runner tags, image digests, or cache keys.
• Artifact path matching is string containment. It will false-positive on comments.

Who should not use this approach? Teams whose CI is only remote templates they cannot fork. Pipelines that are already a DAG and merely look noisy. Anyone hoping a sketching host replaces protected variables or a real lint against the project. Skip it if policy forbids sending the YAML to any machine you do not operate.

After the graph is honest

Commit pipeline.mmd next to .gitlab-ci.yml. Generate it in the same proposed script so the file is cheap to refresh. When the next rewrite “simplifies stages,” the merge request has to update the mermaid. If the arrow count jumps back to the full barrier set, the DAG was flattened again.

The review question is not whether the YAML is shorter. It is whether test:unit still waits on a scan it never reads. Answer that with the table and the graph. Then lint on GitLab, on the project that will actually run the jobs.