nyxd now builds images, is rootless, runs on macOS & Windows

Back in May, Mohamed wrote about getting fed up with the container stack using more RAM than the workloads on his Raspberry Pis, and building nyxd to find out how small a correct, secure OCI stack could be.

I Got Tired of Docker Eating My Raspberry Pi's RAM — So I Built My Own Container Orchestrator

Minimal OCI stack for CVE reduction

Mohamed Zrouga

Mohamed Zrouga

Mohamed Zrouga

Follow

May 16

I Got Tired of Docker Eating My Raspberry Pi's RAM — So I Built My Own Container Orchestrator

#containers #security #raspberrypi #homelab

23 reactions

9 comments

10 min read

I'm Marwa, a platform engineer, and I've been contributing to nyxd alongside Mohamed. A lot has landed since that post: a 1.0 stable release, then 1.1, Dockerfile builds without BuildKit, rootless mode, macOS and Windows clients, and real package repositories.

The original post ended with an unusually honest section: what works, what's half-wired, what's still missing. So this update is structured the same way. We go back through that list and say what happened to each item.

TL;DR

• 1.0 is out, with a written compatibility promise: the /v1 API, CLI flags, on-disk data and log format stay compatible for all of 1.x.
• nyx build builds images from a Dockerfile. No Docker, no BuildKit, no extra binary. RUN steps execute in the same crun sandbox as your containers.
• Rootless: start nyxd as a normal user and it runs everything inside its own user namespace.
• macOS and Windows: a native nyx CLI that drives nyxd in a small Linux VM (vfkit on macOS, WSL2 on Windows).
• nyx compose is a real orchestrator now, not just a YAML parser.
• Signed package repos at pkgs.nyxd.io for apt, dnf/yum, zypper, apk and pacman, plus one-line installers.
• Still two binaries on the host: nyxd and crun.

Try it in a minute

# Linux and macOS
curl -fsSL https://pkgs.nyxd.io/install.sh | sh

# Windows
irm https://pkgs.nyxd.io/install.ps1 | iex

The installer adds the repository for your distro and sets nyxd up as a service for systemd, OpenRC, runit or SysV init. That last part matters on Alpine or a minimal Pi image, where systemd isn't a given. On macOS it sets up the nyx VM, on Windows the WSL2 machine.

Then, the Docker muscle memory works:

nyx run --rm -it alpine sh

--rm and -it landed this week in 1.1.4 / 1.1.5.

Revisiting the "honest list"

✅ Health checks — was: library-ready, not operator-ready

In May, internal/health existed but nothing started it. Now exec / HTTP / TCP probes are wired into the supervisor: they run with each container, run waits for readiness, and an unhealthy container is restarted when its restart policy allows. Restarts use exponential backoff with jitter, capped at 30s.

✅ Compose — was: a parser, not a scheduler

nyx compose up            # finds nyx-compose / docker-compose / compose / podman-compose .yml
nyx compose up --build    # rebuild services that have build:
nyx compose stop          # reverse dependency order
nyx compose down -v       # also remove named volumes, when nothing uses them

It now covers a practical subset of the format: depends_on (list and map form) with sequential start, healthchecks, build:, env_file:, named volumes, deploy.resources.limits mapped to cgroup v2 (memory, cpus, swap, pids, cpuset), ulimits, dns, seccomp_profile and apparmor_profile.

New in 1.1: per-network native bridges. A compose network with bridge and subnet gets its own nyxbr* bridge, its own IPAM, NAT, isolation from other bridges, and its own embedded DNS zone, so services find each other by name. Services that don't declare a network stay on nyxbr0. Still no CNI plugins involved.

✅ Registry auth — was: anonymous Docker Hub pulls only

nyx login registry.example.com     # or reuse your existing docker / podman login
nyx push registry.example.com/team/myapp:1.0
nyx save -o myapp.tar myapp:1.0
nyx load -i myapp.tar              # also accepts docker save / podman save archives

Private registries work for pulls, pushes and compose, and pulls resume if a layer download is interrupted.

✅ Seccomp and cgroups

Seccomp and cgroup v2 limits are enforced, and compose services can bring their own seccomp_profile. A detail worth knowing if you're coming from older nyxd: since 1.0, a memory limit also caps swap (at twice the limit, Docker's default). Before that, a container over its limit swapped instead of getting OOM-killed, which on a Pi with an SD card is the worst of both worlds.

⏳ Still not done, honestly

• Prometheus /metrics: the handler exists, but the default daemon still doesn't serve it.
• sd_notify: nyxd doesn't send READY=1 yet, so keep systemd units on Type=simple (the shipped units already do).
• Compose gaps: no nyx volume ls, depends_on conditions are read but only the order is used, and restart: on-failure:N syntax isn't exposed yet.
• nyxd's socket is not the Docker API. Tools that need docker.sock still need a real Docker socket or DinD; there's a page on that.

The big new pieces

nyx build — Dockerfiles without BuildKit

nyx build -t myapp:1.0 .
nyx build -t myapp:dev -f docker/Dockerfile.dev --build-arg VERSION=1.2 --target runtime .
nyx run myapp:1.0

The Dockerfile parser, layer engine and build cache all live inside nyxd. For each step it mounts overlayfs over the current layers: RUN goes through crun with the same seccomp profile, reduced capabilities and noNewPrivileges as a normal container, while COPY/ADD write straight into the tree from Go. The overlay's upper directory becomes the new layer, and deletions become OCI whiteouts.

Layers are extracted and created in-process now, with no host tar. That's one more external tool off the list. Multi-stage builds, --target, --build-arg, .dockerignore and the cache all work, and it works rootless.

Rootless

nyxd                      # as your normal user: rootless is automatic
nyx run alpine:3.20 id    # uid=0(root) inside, your uid on the host

nyxd re-executes itself inside its own user namespace, and containers run inside that namespace too. Root in a container is your user on the host, never real root. It needs Linux ≥ 5.11 and unprivileged user namespaces. Ubuntu 24.04+ blocks those through AppArmor by default, and the rootless docs cover the one sysctl to change.

macOS and Windows

People asked "can I use this on my laptop?" Now you can:

• macOS: nyx machine runs nyxd + crun in a small Alpine VM on vfkit, and the Mac talks to it over vsock.
• Windows: nyx.exe drives nyxd inside a minimal WSL2 distro.

The CLI is native, but the runtime is always Linux, so compose, bridges and cgroups behave the same as on a Pi. There's a platform matrix showing exactly what works in rootful, rootless, userns-remap, macOS and Windows.

Built to survive a 2AM failure

@itskondrat commented on the first post that a custom orchestrator's real cost shows up at 3AM, when something crashes and there are no docs to search. Fair point, and a lot of the work since then has gone into exactly that:

• Crash recovery: if nyxd is SIGKILLed while containers keep running, the next start re-adopts them, including exec into them, rootless too.
• Bounded logs: rotation by size (3 × 10 MiB per container by default), nyx logs --tail reads from the end, and -f follows across rotations. No more logs quietly filling an SD card.
• Stress testing in CI: a weekly workflow runs 50 containers at once, 200 short-lived ones, a log flood, an OOM kill next to a healthy service, and a daemon crash with 10 containers running. It fails the run if containers aren't re-adopted within 30s, or if daemon memory or open files grow.
• Test coverage gate in CI over unit and end-to-end tests. Writing those tests found and fixed real bugs, such as image prune deleting images still in use, and exec -i hanging after the command exits.
• Docs at nyxd.io, including a get-started page and an OpenAPI explorer for the control API.

A correction to the first post

The first post said the runtime had three external Go modules. That's no longer true, and we'd rather say so. The OCI spec modules are gone (we use our own types), but interactive TTYs and the embedded DNS server brought in others. The direct list today is:

github.com/creack/pty      # nyx run -it / exec TTYs
github.com/miekg/dns       # embedded DNS for service names
golang.org/x/sys
golang.org/x/term
gopkg.in/yaml.v3           # compose

Five direct modules, two indirect. Still small, still auditable in an afternoon, and Dependabot now watches all of them. On the host it's still just nyxd + crun, with nft for port publishing and no /opt/cni/bin unless you choose -net-driver=cni.

About those benchmarks

@fyodorio asked how nyxd compares with Podman on RAM. We promised proper numbers, and we haven't published them yet. What we have done is publish the method: three identical Alpine 3.23 guests under QEMU (Docker, Podman, nyxd), the same workload, the same load generator, and a template for recording every setting. The stress workflow also reports daemon RSS at idle, under load and at peak on every run.

The comparison post comes next, with the numbers and how to reproduce them.

Thank you

Thanks to everyone who commented on the first post. Those questions shaped this release. Thanks also to Sven Nellemann, who set up the documentation site. If you hit a bug, there are issue forms on GitHub, and the docs are open to contributions.

👉 GitHub: https://github.com/zrougamed/nyxd
👉 Docs: https://nyxd.io
👉 Install: curl -fsSL https://pkgs.nyxd.io/install.sh | sh

Questions for the comments:

• If you tried nyxd since May, what broke first?
• Which compose keys are you missing that stop you moving a stack over?
• Rootless or rootful on your edge boxes, and why?
• What should the Docker vs Podman vs nyxd benchmark measure that most benchmarks skip?

nyxd is free for personal and other noncommercial use under the PolyForm Noncommercial License 1.0.0.