Canonical under DDoS: what my Railway logs and uptime say about my real exposure

Why do we assume shared infrastructure "just works" until it stops working for someone else? I spent years living inside that assumption before the Canonical DDoS forced me to actually measure it against my own logs.

Last Wednesday I opened HN and saw "Canonical under DDoS attack" sitting at 178 points. First instinct: scroll past. Second instinct — the one that won — open Railway and figure out exactly how tied I was to the servers someone was hammering at that moment.

The answer was uncomfortable.

Ubuntu DDoS 2025: what happened and why it matters in production

The attack targeted Canonical's distribution infrastructure — mirrors, APT repositories, the network that feeds apt-get update on millions of machines. It wasn't a code breach. Nothing was stolen. It was volumetric: flood the servers until apt stops responding.

For most devs in production that sounds like a "sysadmin problem." Until you count how many times per week your pipelines run apt-get update inside a Dockerfile.

I counted at least 6 different images in my current stack. All of them with apt-get update hardcoded in the build step.

My thesis: indie devs depend on shared public infrastructure way more than we admit out loud, and the Canonical DDoS made that visible all at once. Not because the attack hit us directly — in my case it didn't. But because for the first time in months I actually asked myself what would have happened if it had lasted 48 more hours.

What my Railway logs said when I went looking

First move: pull the Railway build logs for the last 30 days and search for anomalous latency on any step that invokes Ubuntu mirrors.

# Search exported Railway logs for any apt timeouts
grep -E "(apt-get|apt |dpkg)" railway-build-logs-june.txt | \
  grep -E "(timeout|could not|failed|Unable to fetch)" | \
  sort | uniq -c | sort -rn

Result: 11 occurrences of "Unable to fetch" spread across 4 different deployments. Not a single alert had fired. Every single one had failed silently and retried on its own.

That's the part that bothers me. Not that it failed — Railway retries. But I had zero visibility into the fact that I was touching public Ubuntu mirrors that frequently. It was invisible infrastructure.

Then I looked at build times:

# Calculate average time for "RUN apt-get update" step per week
# (data exported from Railway dashboard > Deployments > Build Logs)
awk '/apt-get update/{start=$1} /done/{if(start) print $1-start; start=""}' \
  railway-build-steps-june.txt | \
  awk '{sum+=$1; n++} END {print "Average:", sum/n, "seconds"}'
# Output: Average: 23.4 seconds

23 seconds average for apt-get update. On normal days. During the DDoS, that number would have gone straight to timeout (Railway's default: 10 minutes for the full build before canceling).

If the DDoS had hit while I needed to deploy something urgent on a Saturday at 11pm — exactly the kind of hour where I learned to diagnose production problems back in the day — that apt-get update would have taken down the entire deploy.

The attack surface I never mapped: dependencies that pull Ubuntu without saying so

Here's the part that surprised me most. I went image by image through my Dockerfiles tracking where Ubuntu was sneaking in:

# Image 1: main API
FROM node:20-bookworm-slim
# "bookworm" is Debian, not Ubuntu — OK, doesn't touch Canonical mirrors

# Image 2: processing worker
FROM python:3.11-slim
# Also Debian base — OK

# Image 3: internal admin tool
FROM ubuntu:22.04
# HERE it is. Direct Ubuntu, APT points to archive.ubuntu.com

# Image 4: base for DB migration scripts
FROM ubuntu:20.04
# Another one. Old LTS I never updated because "it works"

Two images directly on Ubuntu. Plus a third that inherits from a custom image I built 8 months ago on top of ubuntu:22.04 and use as an internal base:

# Search my private Railway registry for images that inherit from my ubuntu base
docker image inspect $(docker images -q) --format '{{.RepoTags}} {{.Config.Image}}' \
  2>/dev/null | grep -i "juanchi-base"
# Output: 3 images using juanchi-base:latest as FROM

Total: 5 images that at some point during build or runtime call Ubuntu mirrors. I'd never counted them. I'd never seen a concrete number.

This connects directly to what I wrote about the Linux kernel and vulnerabilities that don't notify distributions: the upstream dependency chain has more nodes than you see on any given day, and every node is a surface.

I simulated what 48 hours of sustained DDoS would have looked like

I can't reproduce the actual attack volume. But I can simulate the most relevant effect for an indie dev: archive.ubuntu.com returning timeouts or 503s during a critical deploy.

Method: redirect the DNS for archive.ubuntu.com inside a local container to a server that doesn't respond, then measure the impact on the build flow.

# Inside a test container with ubuntu:22.04
# Add a fake /etc/hosts entry to simulate downed mirrors
docker run --add-host=archive.ubuntu.com:127.0.0.1 \
           --add-host=security.ubuntu.com:127.0.0.1 \
           ubuntu:22.04 \
           bash -c "time apt-get update 2>&1 | tail -5"

Actual output from my machine:

Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
  Could not connect to 127.0.0.1:80 (127.0.0.1). - connect (111: Connection refused)
Err:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
  Could not connect to 127.0.0.1:80 (127.0.0.1). - connect (111: Connection refused)
Reading package lists... Done
W: Some index files failed to download...
real    0m18.432s

18 seconds to fail. Not instant — it retries several times before giving up. In a CI pipeline with 6 apt steps, that's potentially 108 seconds of build that ends in error even though your code is perfectly fine.

The practical result: in a sustained DDoS scenario against Canonical, my urgent deployments would fail with zero errors in my code. The diagnosis would be confusing because the log would show "build failed" on the dependencies step, not on application code.

This brought me back to my post about supply chain attacks in ML dependencies: the failure vector doesn't always come from code you wrote — it comes from infrastructure you took for granted.

The gotchas I found (and that you probably have in your own stack)

1. "Slim" images don't save you if you build on top of them

node:20-slim uses Debian, sure. But if at any build step you install something with apt-get — tzdata, curl, libvips for sharp — you're touching Debian mirrors that have the exact same dependency on centralized public infrastructure. The domain changes. The pattern doesn't.

2. Railway's cache doesn't always help when you need it most

Railway caches Docker layers. If the apt-get update layer hasn't changed, it won't re-run. Great. But if the Dockerfile changed in anything before that step — which happens constantly in active development — the cache is invalidated and it hits the mirrors again. Right when the system is under stress.

3. apt-get update without --fix-missing fails hard

# This fails hard on degraded mirrors:
RUN apt-get update && apt-get install -y curl

# This at least tries to keep going:
RUN apt-get update --fix-missing && apt-get install -y --no-install-recommends curl

# What you actually want if the mirror might be down:
RUN apt-get update --fix-missing || true && \
    apt-get install -y --no-install-recommends curl 2>/dev/null || \
    echo "WARNING: apt degraded, continuing without optional packages"

The || true is debatable — you're ignoring errors. But for non-critical packages at build time, a deploy with a warning beats a canceled deploy at 11pm on a Friday.

4. Nobody has private Ubuntu mirrors for their indie deploys

Here's the real asymmetry. A big company has Nexus, Artifactory, or an internal mirror. An indie dev has... the same archive.ubuntu.com as everyone else. No buffer layer. When the public mirror fails, it fails for everyone regardless of scale.

You feel this differently than when you're on a large team. In my analysis of bugs Rust doesn't prevent I landed at the same conclusion from a different angle: the tools are optimized for teams with redundancy, not for the indie running solo on Railway with a free Sunday afternoon.

FAQ: Ubuntu DDoS 2025 and impact on indie production

Did the Canonical DDoS actually affect real deploys on Railway or other platforms?

Depends on when you deployed during the incident. Railway uses images that in many cases touch archive.ubuntu.com or Debian mirrors during the apt-get update build step. If the build happened at the peak of the attack and mirrors were degraded, the step could have failed or taken much longer than normal. In my case, logs show 11 failures during the period but none blocked an active deploy — the timing didn't line up. The exposure is real though.

What's the first thing I should check in my Dockerfiles to reduce this dependency?

Count how many times apt-get update appears across your images and what base they're built on. If you're using ubuntu:XX.XX directly, you're a direct client of archive.ubuntu.com. If you're using official language images like node, python, or golang, they normally inherit from Debian, not Ubuntu. That difference matters because they're separate mirror networks.

Does it make sense to set up a private Ubuntu mirror for an indie or small startup?

For a single dev, the overhead doesn't justify the benefit. A full Ubuntu mirror takes between 80 GB and 200 GB depending on architectures. What does make sense: use apt-get install --no-install-recommends to minimize the number of packages you pull, and consider distroless or Alpine images for workloads where you don't need apt at runtime at all.

How long did the Canonical DDoS last and how severe was it?

The incident was reported on Hacker News with 178 points and active discussion. Canonical confirmed the attack on their distribution infrastructure. The exact duration of peak impact isn't publicly documented with hour-level precision, but the HN thread shows reports of slow or inaccessible mirrors over several hours. To simulate the impact on your own stack, the method I used in this post — redirecting the mirror DNS to localhost — is reproducible and gives you a concrete picture of the failure timeline.

Does Alpine Linux solve the public infrastructure dependency problem?

Partially. Alpine uses apk and its own mirrors (dl-cdn.alpinelinux.org), which is separate infrastructure from Canonical. You migrate the dependency, you don't eliminate it. The real advantage of Alpine in this context isn't redundancy — it's size: images are smaller, build times are shorter, and the frequency with which you need to touch the package manager drops significantly. Less surface area is better even if it's not zero.

Does Railway have any protection mechanism against degraded upstream mirrors?

Railway caches Docker layers, which helps if the apt layer hasn't changed. But if the Dockerfile changes — which happens constantly in active development — the cache is invalidated. There's no native mechanism for "fall back to cache if upstream is degraded." That's something you have to implement yourself in the Dockerfile with flags like --fix-missing or conditional build logic.

The invisible dependency doesn't disappear just because we're not measuring it

The moment that changed how I think about Docker wasn't reading documentation — it was a 2015 migration where an app that would have taken 2 days to move took 10 minutes. The magic of "runs anywhere" has a massive asterisk: it assumes that "anywhere" has reliable access to the same public infrastructure you pulled your dependencies from.

The Canonical DDoS didn't break anything for me. My deploys kept running. But it made me count: 5 images with direct or indirect dependency on Ubuntu mirrors. 11 silent failures in 30 days that I had never seen. A simulated failure time of 18 seconds per attempt on downed mirrors.

Those numbers didn't exist before this week. Now they do. And I've already changed two Dockerfiles to use --no-install-recommends and a strategic || true on steps with non-critical packages.

My position: I'm not setting up a private mirror or migrating everything to Alpine this week. But I am adding a weekly check in my Railway logs hunting for failures on apt steps, the same way I already have alerts for application errors. Shared infrastructure is a reality of the indie stack — the problem isn't using it, it's not measuring it.

If you found something similar in your own stack, tell me in the comments. I'm genuinely curious whether the silent failure count is something other devs also weren't seeing.

Original source: Hacker News

This article was originally published on juanchi.dev