Two Proxmox nodes feel like redundancy. One box dies, the other carries on. That is the picture most people have when they join a second machine to the cluster.
Out of the box, it is the opposite: two machines that can each stop the other from working. This post covers why that happens, the small fix that solves it, and the two related traps that make a cluster look haunted.
How quorum works, in one paragraph
A Proxmox cluster uses Corosync to agree on who is alive. Each node gets one vote, and the cluster is quorate when more than half the votes are present. Without quorum, the cluster filesystem at /etc/pve goes read-only. You can't start guests, change configs or create backups, and nothing in the UI saves.
The trap: 1 vote out of 2 is not a majority
Now do the arithmetic for two nodes. Lose either one, and the survivor holds 1 vote out of 2. That is exactly half, and half is not more than half. So the survivor freezes.
That is the part that catches people. The node that is still up, still powered and still on the network is the one that stops working, because it has no way to know whether the other node is dead or just unreachable. Refusing to write is the safe choice for the cluster, and it is a very annoying choice for you.
You can see the vote count at any time:
pvecm status
The fix: a third vote that isn't a third server
A QDevice is a tiny external voter. It can run on a Raspberry Pi, a NAS container or any always-on Linux box that isn't one of the cluster nodes. It gives the cluster a third vote without a third hypervisor.
With three votes, losing one node leaves 2 out of 3. That is more than half, so the survivor stays quorate and keeps working.
The setup is three installs and one command:
# On the external box (Debian/Ubuntu/Raspberry Pi OS):
apt install corosync-qnetd
# On EVERY cluster node:
apt install corosync-qdevice
# On ONE cluster node (root SSH to the QDevice box must work):
pvecm qdevice setup 192.168.10.5
# Check: you should see 3 total votes and "Qdevice" listed
pvecm status
Replace 192.168.10.5 with the address of your own external box.
Gear mentioned (As an Amazon Associate I earn from qualifying purchases. Affiliate links.) Raspberry Pi · All the gear the books recommend
Where the QDevice must not live
Don't run the QDevice as a VM or container on one of the cluster nodes. It is tempting, because the hardware is already there. But if that node goes down, it takes its own vote and the QDevice's vote with it, and you are back to 1 vote out of 3 on the survivor. Separate hardware is the whole point.
| QDevice placement | One node dies |
|---|---|
| VM or container on a cluster node | That node takes two votes with it; no quorum |
| Separate always-on box | The third vote survives; the other node keeps quorum |
The related trap: a node on Wi-Fi that keeps flapping
Even with enough votes, a cluster can look broken if one member keeps dropping in and out. The usual cause is Wi-Fi.
Corosync is extremely sensitive to latency and jitter. It wants a stable, low-latency link: single-digit milliseconds, consistently. Wi-Fi delivers great average speeds and terrible consistency. A microwave, a neighbor's channel change or a roaming decision adds 50 to 300 ms of jitter for a few seconds. Corosync sees missed tokens, decides the node is gone, and the cluster reshuffles. A few seconds later it is back. Then it is gone again.
The symptoms look random:
- The node shows a red X, then turns green again
-
/etc/pveis briefly read-only - Backup jobs fail with "cluster not ready"
- The logs fill with lines like
[TOTEM] A processor failed, forming new configuration
Check for it:
journalctl -u corosync --since "24 hours ago" | \
grep -Ei "failed|new membership|retransmit"
corosync-cfgtool -s # link status per node
ping -c 100 -i 0.2 <other-node> | tail -2 # look at mdev/max, not avg
Look at the mdev and max numbers from the ping, not the average. A good average with a big max is exactly the jitter Corosync hates.
The fix isn't a Corosync tuning trick. It's a cable. If the node physically can't be wired, use a powerline or MoCA adapter (still better than Wi-Fi), or take it out of the cluster and run it standalone. A standalone node you manage separately is far less trouble than a cluster member that drops out twice an hour.
If your nodes have two network ports, give Corosync a second link (link1) on a separate network or a direct cable between the nodes. Corosync fails over between links automatically.
Gear mentioned (As an Amazon Associate I earn from qualifying purchases. Affiliate links.) Powerline or MoCA Ethernet adapter · All the gear the books recommend
When you're stuck without quorum right now
If one node of two is dead and you need the survivor working immediately, there is an emergency lever:
pvecm expected 1
This tells the survivor to accept being alone. It is an emergency lever, not a fix. Only use it when you are sure the other node is truly off, because two nodes that each think they are the whole cluster can write conflicting configs. Once the crisis is over, the real fix is still the QDevice.
Scripts: wait for quorum instead of failing halfway
If you script changes on a cluster with a flaky member, check quorum before each write and retry, instead of letting the script die partway through a batch:
until pvecm status | grep -q "Quorate:.*Yes"; do
echo "waiting for quorum..."; sleep 10
done
pct destroy 123 --purge
The loop waits until pvecm status reports the cluster as quorate, then runs the write. Wrap each destructive or config-changing step this way and a brief membership blip costs you a pause, not a half-finished job.
Checklist
- Every cluster node is wired, ideally on the same switch
- Two-node clusters have a QDevice on separate hardware, never on a cluster node
-
pvecm statusshows the expected vote count (3 for two nodes plus a QDevice) - Corosync logs show no membership changes over the past day
- Nodes with two ports give Corosync a second link (
link1) - Scripts check for quorum before each write
- You know the
pvecm expected 1lever, and when not to pull it
Drafted with AI help. I edited it, and it's based on my own setup.
This post is adapted from Chapter 2 of my ebook Homelab the Right Way: Proxmox, OPNsense & Home Assistant. The free sample is a full chapter; the whole book covers the cluster build, storage, backups and the network around it. All 18 of my homelab and Android ebooks are in one $89 bundle (code RIGHTWAY20 takes 20% off through Oct 31). Get the free weekly email: https://chartmann1590.github.io/homelab-the-right-way/newsletter/
The 60-second version is on YouTube: https://youtube.com/shorts/v00ZPYLWcfM
