You can have enterprise storage, a healthy host and a filesystem doing exactly what it was designed to do, and still lose a node to one network interruption. This is what a firewall racking job did to my Sia hostd node, and what it taught me about how much interruption iSCSI will actually absorb before the filesystem underneath runs out of options.
It started with something that looked like a failure and turned out to be me misreading my own setup. Twenty-four hours later it became the real thing.
// the setup
The hostd node runs as a VM on pro4, one of the Proxmox hosts, on the workload VLAN. Storage comes from TrueNAS over iSCSI — a zvol presented as /dev/sdb, formatted XFS, mounted at /mnt/sia-data, with /var/lib/hostd symlinked into it. hostd itself is a single binary handling contracts, storage proofs and peer connections.
iSCSI over a routed network is convenient. It is also unforgiving. Convenient because the disks live where the disks live, the compute lives where the compute lives, and neither has to know. Unforgiving because the amount of interruption it will tolerate is a fixed number in a config file, and most of us have never looked it up.
I hadn't.
// act 1 — the false alarm
The morning after the migration, the hostd GUI wouldn't load. My first assumption was a crash, and the logs backed it up — a startup attempt with failed to create data directory, which reads alarming at seven in the morning.
It took about thirty seconds to establish I was wrong. hostd was already running. Process up, port 9980 bound and listening. My systemctl start had found a live process and exited without doing anything. The GUI had been reachable the whole time; I had the browser pointed at the management VLAN address rather than the workload one.
So: operator error, not a node failure. Lesson one, filed and then almost immediately forgotten — verify before you assume.
// act 2 — the real one
Twenty-four hours later I was racking the new border firewall. During the physical install it was down for roughly twenty minutes. That was planned and I was expecting it. What I had not fully accounted for was that the iSCSI path to TrueNAS ran through it.
The sequence:
- The network path to TrueNAS dropped when the firewall went dark.
- Five seconds later, with no reply to its NOP-out ping, the initiator declared the connection failed.
- The session went into recovery and I/O queued, waiting for the session to come back.
- Two minutes in, it hadn't. SCSI commands were failed back to the application and
/dev/sdbwent away. - XFS saw write errors on a device that no longer existed and shut down the journal to protect the filesystem.
- hostd was mid-write.
database disk image is malformedonhostd.sqlite3.
journalctl told the whole story in two lines — write I/O errors, followed immediately by XFS shutting itself down. hostd had crashed cleanly enough, but the SQLite write-ahead log was in an uncertain state and the database was 41 GB.
// the two numbers I should have known
Here is the part I had wrong, and had been wrong about for as long as I'd been running this.
I assumed iSCSI had no tolerance at all — that a dropped path meant an immediate dead device. It doesn't. There are two timers, they do different jobs, and the gap between them is where the fault tolerance lives:
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.session.timeo.replacement_timeout = 120
Those are the stock defaults, unchanged on my host. The initiator pings the target every five seconds. If a ping goes five seconds without a reply, the connection is declared failed — that's the first number, and it's the one I'd half-remembered as "the timeout".
It isn't the one that matters. replacement_timeout is how long the session is given to re-establish itself before SCSI commands get failed back to the application. Default 120 seconds. For two full minutes, I/O queues and waits. If the path returns inside that window, the writes land and nothing upstream ever knows there was a problem.
So iSCSI is not fragile. It ships with two minutes of patience.
I was gone for twenty.
That reframes the incident entirely. Nothing failed — not the initiator, not XFS, not SQLite. Every layer did exactly what it was built to do, in order, on a timescale that was documented and that I could have read in thirty seconds any time in the previous year. The outage was ten times longer than the tolerance, and after that there is nothing left for a filesystem to do except stop.
// the recovery
Methodical, and mostly waiting. Once the firewall was back and the path restored:
umount -l /mnt/sia-data
Lazy unmount, then remount — XFS replayed the journal automatically on the way back up.
sqlite3 /var/lib/hostd/hostd.sqlite3 "PRAGMA integrity_check;"
Then wait. On a 41 GB database the integrity check ran for about three hours. The result, when it finally came back, was one word: ok.
systemctl start hostd. The GUI came up, contracts resumed, and RHP4 RPC successes started streaming in the logs within minutes. Nothing was lost.
// what I'd make instinctive
Three things from this.
Stop the service and unmount before touching anything on the storage path. Routing changes, firewall reboots, link reseats, a switch upgrade — if the path between the VM and the iSCSI target is in scope, hostd stops and /mnt/sia-data unmounts first. It costs a minute. The alternative cost me three hours of integrity check and a morning of not knowing whether the database had survived.
Know your replacement_timeout before you need it, not after. Two minutes is the default, not a law — it's one line in iscsid.conf and some distributions and appliances ship it differently. Run iscsiadm -m node -o show | grep -E 'replacement_timeout|noop_out' on anything you care about and write the answer in the runbook next to the maintenance window. A twenty-minute job against a two-minute tolerance is a decision, as long as you know you're making it.
SQLite's WAL is resilient, but only because XFS was. The write-ahead log protected the database and the XFS journal protected the filesystem. Both did their jobs. The three-hour integrity check wasn't a symptom of damage — it was the cost of confirming the safety nets had held, on a database large enough that confirming takes three hours.
The runbook now has a mandatory pre-change step: stop hostd, unmount the iSCSI volume, then proceed. Every time.
// closing
The ARP race piece from last week was the same shape at a different layer — a change that verified fine at one point and broke at a boundary nobody had checked. This one is the same lesson with a number attached. The maintenance window was planned, the outage length was known in advance, and the piece of information that would have made it a non-event was sitting in a config file I'd never opened.
The tolerance exists. It's just shorter than most maintenance windows, and nothing warns you when you're about to spend it.
