When infrastructure was physical

I was once happy — properly happy — about a load balancer.

It was an NEC appliance, a proper pair in active and hot standby, and what made it a relief was something small and specific. Until then the web tier ran behind a software load balancer I'd wedged onto one of the servers. It did the job — but when I took a backend out of rotation, it cut the connections still open along with it, and somewhere a user watched their browser fill with an error. I hated that out of all proportion to what it was. What I wanted, badly, was a balancer that would let the requests already in flight finish before it let a server go. When one finally arrived — draining each backend quietly before it dropped it — I was happier than a piece of network hardware has any right to make me.

The service it fronted ran on three rackmounted machines — Web1, Web2, and a MySQL box — the first rackmount servers I ever put my hands on. They were where my working life started.

I don't think I've felt that exact happiness about infrastructure in a long time. I want to describe what it was, because most of it is gone now, and I'm not certain the trade was free.

Back then, infrastructure was something you committed to before it existed. You chose a datacenter. You worked out the weight your racks were allowed to bear and the power you would draw, signed a contract for that power, and guessed — months ahead — how much you would actually need. Bandwidth was expensive, so you forecast your traffic and paid for a pipe sized to a number you had half made up. Every decision was a bet placed before the thing it was betting on arrived. You paid, in money and floor space and your own time, for load that hadn't shown up yet.

Then the machines had to physically get there. I once had an Oracle Exadata that wouldn't fit through the datacenter door; we widened the opening to bring it in and reinforced the floor where it would stand. I kept its load average pinned as low as I could manage — the kind of attention you give something you feel responsible for. You don't forget carrying that much weight into a room.

Some of the fleet was scavenged. A stack of blades left over from a project that hadn't worked out got a second life as something else entirely — hardware outliving the idea it was bought for. Other machines existed only because of limits: a dense, low-power blade chassis that made sense only because a fully loaded rack could pull thirty-eight kilowatts, and power and floor space were what you ran out of first.

Nobody could tell me how much data we'd have in three years, and there was no S3 to wave the question away. So I read everything I could and landed on Isilon, because it could grow while it stayed online. There was no one to ask; you found these things yourself, or you didn't find them at all.

It wasn't all the thrill of it, though there was thrill. It was also a long argument with failure. Most failures were almost friendly — a disk lamp on a RAID6 array you could practically ignore, or half a power feed dying while the redundancy quietly absorbed it. You learned not to fear those.

The ones that got you were the unplanned ones. I remember a router that started failing with nothing behind it — no second unit, nothing to fail over to, nothing to do but the thing you didn't want to do. I closed my eyes, pulled the cable, swapped in the spare, and waited to learn whether I had just made it worse. Those were the ones that wore on you. There was no pager rotation. You were the redundancy.

None of this was ever quite mine alone. I wrote the software, but the machines I ran it on were built and racked by someone else — we had one infrastructure engineer, and the iron was his before it became mine. And someone above the two of us decided to turn a pair of engineers loose on that much expensive equipment in the first place. I didn't think of any of it as a gift while it was happening. It was one.

The cloud took all of this away, and I'm mostly grateful. I don't miss forecasting bandwidth, or widening doorways, or pulling cables with my eyes shut. Capacity is elastic now — it comes when you ask and goes when you stop, and no one bets the floor on a guess.

But something left with it. When infrastructure was physical, it was also yours, in a way a call to someone else's datacenter never quite is — something that could thrill you and wear you down in the same week, because it was close enough to put your hands on. I wouldn't go back. I just remember being happy about a load balancer, and I notice I don't feel that about anything anymore.

Built with Claude (Opus).