Why pgrep Can't Detect cloud-init's Apt Races on DigitalOcean

This article was originally published on Jo4 Blog.

If you've ever written a wait loop that polls pgrep apt-get before running your own apt-get, congratulations: you have a race condition. We had one too. On DigitalOcean's Ubuntu 24.04 droplets, our cold-boot setup script would intermittently die on the apt frontend lock — even though we'd "carefully" waited for apt to be idle. Here's why our defensive wait loop was lying to us, and the one line that actually fixed it.

The Setup

We run a small observability droplet — Prometheus + Grafana behind nginx, called jo4-impress. It's bootstrapped from a GitHub Actions workflow that SCPs impress/do-setup/ to the freshly-provisioned droplet and runs setup-impress.sh. The script installs Docker via the convenience script (curl -fsSL https://get.docker.com | sh), then brings up the compose stack.

Because we knew Ubuntu's unattended-upgrades and the apt-daily timers fire shortly after boot, we put a pgrep-based wait loop in front of the Docker install. The idea: poll until no apt-get is running, then run our own. Belt-and-braces. What could go wrong.

The Problem

On a fresh droplet, the script intermittently failed with the classic:

E: Could not get lock /var/lib/dpkg/lock-frontend.
   It is held by process 2746 (apt-get)

The frustrating part: our wait loop reported "no apt-get running" two seconds before this error. We weren't ignoring it. We were checking. And yet PID 2746 — an apt-get we never saw — grabbed the lock the instant our curl get.docker.com | sh invoked its own apt-get update.

Here's the timeline pulled from /var/log/apt/history.log on the droplet (workflow run 26424379616):

PID 2746 was spawned by install-dotty-agent, a DigitalOcean cloud-init vendor script. It started in the four-second gap between install-do-agent finishing and our curl | sh racing into apt-get update. Our pgrep loop caught the quiet moment between two vendor scripts and concluded — correctly, in the instant it sampled — that apt was idle. By the time we acted on that conclusion, the next vendor script had already launched.

The Two Sources of Contention

This is where it gets interesting. On a DigitalOcean Ubuntu 24.04 droplet at first boot, there are two completely independent things competing for the apt lock, and most defenses only address one of them.

(1) The systemd timer units. These are the ones every "first-boot apt fix" tutorial talks about:

• unattended-upgrades.service
• apt-daily.service, apt-daily.timer
• apt-daily-upgrade.service, apt-daily-upgrade.timer

They fire on a randomized schedule shortly after boot. systemctl stop + systemctl mask handles them. This is the well-known half of the problem.

(2) cloud-init's modules-final → config-scripts_vendor stage. This is the half nobody warns you about. DigitalOcean ships two vendor scripts that cloud-init runs during this stage:

• install-do-agent — installs the DigitalOcean monitoring agent
• install-dotty-agent — installs droplet-agent (the web-console SSH bridge)

Both shell out to apt-get install … directly. They are not systemd units. They don't appear in systemctl list-units. Masking the apt timers does precisely nothing to them, because they were never running through those timers in the first place.

On the run captured above, install-do-agent ran 23:48:29–23:49:41 (71.8 seconds total) and install-dotty-agent ran 23:49:41–23:49:53 (12.2 seconds). Between them: a short window where pgrep apt-get returns empty — but cloud-init is very much still going to launch the next one.

Why the pgrep Loop Failed

The pgrep approach has a structural flaw, not a tuning flaw. Let's be explicit:

# The old wait loop (approximately)
for i in $(seq 1 60); do
  if ! pgrep -x apt-get >/dev/null; then
    echo "apt is free, proceeding"
    break
  fi
  sleep 5
done
apt-get install -y docker-ce  # races whatever cloud-init launches next

The check answers one question: is there an apt-get process running **right now? That's a point-in-time observation. It tells you nothing about whether something about to be launched will run an apt-get in the next few seconds.

Cloud-init's vendor-scripts stage is a sequence. While it's executing the sequence, the gaps between individual apt-get invocations are normal — script setup, package downloads, post-install hooks — and routinely large enough to make pgrep look clean. The wait loop is sampling a process state when it should be checking a higher-level lifecycle state: "is cloud-init done yet?"

The systemd-timer half of the problem made this look like a tuning issue ("just sleep longer", "increase the poll count"). It wasn't. No amount of polling helps when the next contender hasn't been spawned yet.

The Fix

Cloud-init ships a command that answers the actual question:

cloud-init status --wait

It blocks until cloud-init reaches its done state — vendor scripts inclusive. Once it returns, every install-*-agent invocation has completed and won't be spawning more apt-gets. We can install Docker without surprise contenders.

The diff that fixed setup-impress.sh:

- # Wait until any boot-time apt is finished.
- for i in $(seq 1 60); do
-   if ! pgrep -x apt-get >/dev/null; then
-     break
-   fi
-   sleep 5
- done
+ # Defense-in-depth: silence the systemd timer units...
+ APT_UNITS=(
+   unattended-upgrades.service
+   apt-daily.service apt-daily.timer
+   apt-daily-upgrade.service apt-daily-upgrade.timer
+ )
+ systemctl stop "${APT_UNITS[@]}" 2>/dev/null || true
+ systemctl mask "${APT_UNITS[@]}" 2>/dev/null || true
+
+ # Load-bearing: block until cloud-init's vendor scripts finish.
+ echo "⏳ Waiting for cloud-init to finish (DO vendor scripts release apt)..."
+ cloud-init status --wait
+
  echo "📦 Installing Docker via convenience script..."
  curl -fsSL https://get.docker.com | sh
  systemctl enable --now docker
+
+ # Lift the masks once Docker is in.
+ systemctl unmask "${APT_UNITS[@]}" 2>/dev/null || true
+ systemctl enable --now apt-daily.timer apt-daily-upgrade.timer 2>/dev/null || true
+ systemctl enable --now unattended-upgrades.service 2>/dev/null || true

Two things to notice. First: cloud-init status --wait is doing the load-bearing work. It's the one line that closes the race. Second: we still mask the apt timers, but only as defense-in-depth against the scenario where cloud-init has finished but a timer fires inside the Docker-install window. We re-enable them at the end so the droplet's normal patching cadence resumes.

The full context lives in impress/do-setup/setup-impress.sh, commented inline so the next person who reads it doesn't have to re-derive any of this.

Lessons Learned

• pgrep answers "is X running now?" — not "will X run in the next ten seconds?" Any defensive wait built around pgrep has this blind spot. Use it for "wait for this specific PID to exit", not "wait for a class of process to be permanently quiet."
• On Ubuntu 24.04 cloud images, there are two boot-time apt actors, not one. Systemd timers and cloud-init vendor scripts run independently. Masking the timers and ignoring cloud-init buys you the silence right up until cloud-init speaks.
• cloud-init status --wait is the right primitive. It encodes the lifecycle question ("is cloud-init done?") that your script actually wants the answer to. No polling, no false positives, no off-by-a-few-seconds.
• Belt-and-braces is fine; one of them has to actually be load-bearing. We kept the systemd masks because they cost us nothing and close a real (if narrower) window. But the masks alone never fixed the bug — cloud-init status --wait did.
• When an intermittent bug has a four-second window, the logs from one bad run are worth more than a week of staring at the code. /var/log/apt/history.log on the failing droplet handed us the answer in the form of PID 2746 and an exact second. The fix took ten minutes once we had it.

Hit a similar cloud-init-vs-apt race? Drop the timeline in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.