Stop Spinning Up Full VMs for Every Test Lab: Practical systemd-nspawn Containers on Linux

You need a throwaway Debian box to try a package upgrade. Or a clean Fedora root to reproduce a packaging bug. Or a second "machine" on the same host so you can break networking without touching production.

A full VM works — and costs you another kernel, another bootloader, another virtual disk, and minutes of cold-start time.

systemd-nspawn is the other option: a chroot with real namespaces. Same host kernel, full userspace OS tree, PID/IPC/UTS isolation, optional private networking and user namespaces, managed like any other systemd unit through machinectl.

This is not "replace Podman for every app container." Application containers (OCI images, Quadlet units, auto-updates) stay the right tool for single services. nspawn shines when you want a whole OS — systemd as PID 1, package manager, multi-service stack — without virtualizing hardware.

What you get (and what you do not)

From the man page, nspawn virtualizes:

• the filesystem hierarchy
• the process tree
• IPC
• hostname / domain name

It deliberately locks down host-affecting interfaces: /sys and /proc/sys are read-only from the container, the host clock and host NICs cannot be reconfigured from inside, device nodes cannot be created, modules cannot be loaded, and the host cannot be rebooted from the guest.

Important safety note from upstream: that sandbox is weak without user namespaces. Untrusted workloads should always run with --private-users / -U (the systemd-nspawn@.service template enables this by default when the kernel supports it).

Boundary check:

Packages

On Debian/Ubuntu:

sudo apt update
sudo apt install systemd-container debootstrap

systemd-nspawn, machinectl, and the systemd-nspawn@.service template ship in systemd-container. debootstrap builds Debian/Ubuntu root trees.

Enable the machine registration target so boot-time containers can start:

sudo systemctl enable --now machines.target
sudo systemctl enable --now systemd-machined.service

Build a Debian root under /var/lib/machines

machinectl looks for images primarily in /var/lib/machines/. Machine names must be valid hostnames: ASCII letters, digits, and hyphens only (underscores are rejected).

MACHINE=deb-lab
sudo mkdir -p /var/lib/machines

# Include dbus + PAM/NSS systemd bits so machinectl shell/login work cleanly
sudo debootstrap \
  --include=dbus,libpam-systemd,libnss-systemd,systemd-resolved,iproute2,iputils-ping \
  bookworm \
  "/var/lib/machines/${MACHINE}" \
  http://deb.debian.org/debian

Why those extra packages? Debootstrap does not pull recommended packages by default. Without dbus and the systemd PAM/NSS pieces, a booted container often cannot talk to the host's machine manager the way you expect.

Set a root password (shell mode — no --boot yet):

sudo systemd-nspawn -D "/var/lib/machines/${MACHINE}" passwd

Or drop straight into a shell:

sudo systemd-nspawn -D "/var/lib/machines/${MACHINE}"
# inside: passwd ; exit

Confirm the OS identity file exists (nspawn requires /etc/os-release or /usr/lib/os-release before boot):

sudo cat "/var/lib/machines/${MACHINE}/etc/os-release"

First boot: interactive vs managed

Interactive console boot

sudo systemd-nspawn -b -D "/var/lib/machines/${MACHINE}"

-b / --boot searches for an init and runs it as PID 1. Log in as root. From inside the container, poweroff shuts it down cleanly. From the attached console, Ctrl+] pressed three times quickly detaches nspawn itself.

Managed service boot (preferred for daily use)

sudo machinectl start "${MACHINE}"
sudo machinectl status "${MACHINE}"
sudo machinectl list

machinectl start instantiates systemd-nspawn@deb-lab.service. That template applies different defaults than a bare CLI invoke:

Open a root shell without a full getty login:

sudo machinectl shell root@${MACHINE}

Or a login prompt:

sudo machinectl login ${MACHINE}

Host-side journal for that machine:

sudo journalctl -M "${MACHINE}" -e

Resource view:

systemd-cgtop
systemd-cgls -M "${MACHINE}"

Persistent settings: .nspawn files

Do not fork the template unit for every tweak. Put durable options in:

/etc/systemd/nspawn/<machine>.nspawn

Example for a lab box with port publish, tighter filesystem policy, and explicit private networking:

# /etc/systemd/nspawn/deb-lab.nspawn
[Exec]
Boot=yes
PrivateUsers=yes
# Optional: drop a capability you never need in this lab
DropCapability=CAP_SYS_MODULE

[Files]
# Keep the image writable for apt; use Volatile=overlay for disposable runs instead
ReadOnly=no
# Example host bind for shared packages or test fixtures
BindReadOnly=/var/cache/lab-fixtures:/fixtures

[Network]
Private=yes
# Required workaround when using .nspawn with systemd-nspawn@.service:
# restate veth even though the unit already passes --network-veth
VirtualEthernet=yes
# Publish container :80 on host TCP 8080 (non-loopback interfaces only)
Port=tcp:8080:80

Notes that bite people in production:

1. Privileged keys (Bind=, Capability=, PrivateUsers=, …) only fully apply from /etc/systemd/nspawn/ or /run/systemd/nspawn/. Files next to the image are partially ignored for safety.
2. With the template's --settings=override, some network keys interact poorly unless you restate VirtualEthernet=yes in the .nspawn file (documented Arch/systemd issue).
3. machinectl remove can delete a settings file under /etc/systemd/nspawn/ in some versions — keep a copy in config management.
4. Port mapping intentionally skips the host loopback. curl 127.0.0.1:8080 hits the host, not the container. Use a real host address or test from another machine/namespace.

Inspect / edit settings:

sudo machinectl cat deb-lab
sudo machinectl edit deb-lab

Networking that actually works

Default private veth (machinectl)

Host side: ve-deb-lab (truncated with an altname if the name is long).

Guest side: host0.

If systemd-networkd runs on the host, the shipped unit /usr/lib/systemd/network/80-container-ve.network matches ve-*, runs a DHCP server toward the container, and can enable IP masquerade. Inside a networkd-enabled guest, 80-container-host0.network DHCP-clients host0.

Minimal host enablement when the rest of the host still uses NetworkManager or ifupdown:

sudo systemctl enable --now systemd-networkd

networkd only manages interfaces it has .network files for — existing host NICs stay alone unless you write units for them.

Verify:

ip -br link show type veth
ip -4 addr show ve-deb-lab
sudo machinectl shell root@deb-lab /sbin/ip -br a
ping -c2 <container-ip>

Forwarding / NAT caveats

networkd may install a masquerade rule for the container subnet, but filter FORWARD policy is still yours. On an nftables host, allow the veth path explicitly, for example:

sudo nft add table ip filter 2>/dev/null || true
sudo nft 'add chain ip filter forward { type filter hook forward priority filter; policy drop; }' 2>/dev/null || true
sudo nft add rule ip filter forward iifname "ve-*" oifname "eth0" accept
sudo nft add rule ip filter forward iifname "eth0" oifname "ve-*" ct state established,related accept
# DHCP to containers
sudo nft add rule ip filter input iifname "ve-*" udp dport 67 accept

Adjust eth0 to your real uplink name. Keep these rules in your normal nftables config, not one-off memory.

Host networking (share the host stack)

For simple command containers where you do not want a private netns:

# /etc/systemd/nspawn/deb-lab.nspawn
[Network]
VirtualEthernet=no

Now the container sees the host's interfaces and addresses. Fine for trusted package builds; poor isolation for anything that binds ports or rewrites routes.

Bridge or zone

• --network-bridge=br0 / Bridge= — attach the host end of the veth to an existing bridge (vb- prefix).
• --network-zone=lab / Zone= — let nspawn/networkd maintain a vz-lab bridge shared by several containers.

MACVLAN / IPVLAN

--network-macvlan=eth0 puts the container on the LAN with its own MAC (mv-eth0 naming). Useful when the lab must look like a physical peer. Remember the host cannot talk to macvlan children on the same parent without a second macvlan on the host side.

Resource limits (cgroup v2)

Treat the container unit like any other service:

# Soft reclaim pressure + hard stop
sudo systemctl set-property systemd-nspawn@deb-lab.service \
  MemoryHigh=1G \
  MemoryMax=2G \
  CPUQuota=200%

That writes drop-ins under /etc/systemd/system.control/. Temporary only:

sudo systemctl set-property --runtime systemd-nspawn@deb-lab.service CPUQuota=100%

Confirm with systemctl show systemd-nspawn@deb-lab.service -p MemoryMax -p CPUQuota and systemd-cgtop.

Ephemeral and volatile modes

Three related knobs, different tradeoffs:

Ephemeral example:

sudo systemd-nspawn -b -x -D /var/lib/machines/deb-lab
# changes vanish when the container exits

Btrfs/XFS reflinks make snapshot modes cheap; plain ext4 still works but copies more data.

Clone a durable second image when you want a named fork instead of a temp snapshot:

sudo machinectl clone deb-lab deb-lab-try2
# hostname / machine-id are NOT rewritten — fix them inside the clone
sudo systemd-nspawn -D /var/lib/machines/deb-lab-try2 \
  hostnamectl hostname deb-lab-try2

User namespaces without the footguns

Managed containers default to -U when available (--private-users=pick --private-users-ownership=auto). That maps container UID 0 to an unprivileged high range on the host.

Implications:

• Host paths bind-mounted into the container show up as "nobody" / odd UIDs unless you plan ownership.
• After a chown-style shift, keep using private users consistently or shift back with an explicit range starting at 0 (see Arch Wiki / man page) before abandoning userns.
• Nested containers need extra delegated ranges (PrivateUsersDelegate= on newer systemd) — out of scope for a first lab, but know the knob exists.

Enable at boot

sudo machinectl enable deb-lab
sudo machinectl start deb-lab   # if not already running

This enables systemd-nspawn@deb-lab.service. Disable with machinectl disable deb-lab.

Daily operator cheat sheet

machinectl list
machinectl list-images
machinectl status deb-lab
machinectl shell root@deb-lab
machinectl poweroff deb-lab     # clean shutdown (SIGRTMIN+4 to PID 1)
machinectl reboot deb-lab
machinectl terminate deb-lab    # hard kill
machinectl copy-to deb-lab ./payload.tar.gz /root/
machinectl copy-from deb-lab /var/log/apt/history.log ./
journalctl -M deb-lab -u ssh.service

Cleanup leftovers after a killed session (newer systemd):

sudo systemd-nspawn --cleanup -M deb-lab

Remove an image you no longer need:

sudo machinectl poweroff deb-lab
sudo machinectl remove deb-lab

Minimal end-to-end lab script

#!/usr/bin/env bash
set -euo pipefail

MACHINE=${1:-deb-lab}
ROOT=/var/lib/machines/${MACHINE}

sudo apt-get install -y systemd-container debootstrap
sudo systemctl enable --now machines.target systemd-machined.service systemd-networkd.service

if [[ ! -e ${ROOT}/etc/os-release ]]; then
  sudo debootstrap \
    --include=dbus,libpam-systemd,libnss-systemd,systemd-resolved,iproute2,iputils-ping,openssh-server \
    bookworm "${ROOT}" http://deb.debian.org/debian
  # empty root password for lab-only; change immediately on shared hosts
  sudo systemd-nspawn -D "${ROOT}" bash -c 'echo "root:root" | chpasswd'
fi

sudo mkdir -p /etc/systemd/nspawn
sudo tee /etc/systemd/nspawn/${MACHINE}.nspawn >/dev/null <<EOF
[Exec]
Boot=yes
PrivateUsers=yes

[Network]
Private=yes
VirtualEthernet=yes
Port=tcp:2222:22
EOF

sudo systemctl set-property systemd-nspawn@${MACHINE}.service MemoryMax=2G CPUQuota=200%
sudo machinectl start "${MACHINE}"
sudo machinectl status "${MACHINE}"
echo "Shell: sudo machinectl shell root@${MACHINE}"
echo "SSH via published port on a non-loopback host IP: ssh -p 2222 root@<host-ip>"

Treat the default password as a lab convenience only. On any shared host, set a real password or install SSH keys and disable password auth inside the container.

Failure modes worth knowing

1. Invalid machine name — rename; no underscores.
2. machinectl shell hangs / fails — install dbus + libpam-systemd in the guest; ensure the guest actually booted with systemd.
3. No outbound network — host networkd not running, missing FORWARD accepts, or ip_forward disabled (sysctl net.ipv4.ip_forward).
4. Port publish "works" but localhost fails — by design; nspawn skips loopback for -p/Port=.
5. Permission chaos on binds — user namespace UID shift; use BindReadOnly= for host content or align ownership deliberately.
6. Untrusted code without -U — treat as nearly equivalent to root on the host for escape purposes.

Rollback

# Stop and disable
sudo machinectl poweroff deb-lab || true
sudo machinectl disable deb-lab || true

# Remove unit property drop-ins
sudo systemctl revert systemd-nspawn@deb-lab.service 2>/dev/null || true
sudo rm -rf /etc/systemd/system.control/systemd-nspawn@deb-lab.service.d

# Remove settings + image
sudo rm -f /etc/systemd/nspawn/deb-lab.nspawn
sudo machinectl remove deb-lab
# or: sudo rm -rf /var/lib/machines/deb-lab

No kernel modules were loaded, no permanent sysctl was required for the basic lab. Only your nftables FORWARD/NAT additions need a matching delete if you added them by hand.

When to reach for something else

• Single process, OCI image, registry workflow → Podman/Docker (and Quadlet if you want systemd-native units).
• Different kernel, Secure Boot guest, full device model → libvirt/QEMU.
• Maximum untrusted isolation on the same kernel → still consider a VM; nspawn userns is good, not magical.
• Orchestrated fleets → Kubernetes/Nomad; nspawn is a host-local machine manager.

References

• systemd-nspawn(1) — container runtime options, volatile/ephemeral modes, networking flags
• systemd.nspawn(5) — .nspawn settings file format
• machinectl(1) — start/shell/enable/clone/remove workflow
• systemd.resource-control(5) — MemoryMax=, CPUQuota=, etc.
• ArchWiki: systemd-nspawn — debootstrap examples, veth/NAT notes, userns tips
• Debian man: systemd-nspawn — distro-packaged reference
• debootstrap(8) — building Debian rootfs trees

Full OS. Same kernel. Unit-file lifecycle. Once the root tree exists, machinectl start is closer to flipping on a service than waiting on a hypervisor — and that is exactly when a "test VM" stops being worth the weight.