A cleanup script, an empty variable, and my side project's files gone at 6:04 PM. The fix wasn't skill — it was blast radius, snapshots, and restore drills on Krova Cloud.
The command was a cleanup script. The variable was empty. The path resolved somewhere it absolutely should not have, and by 6:04 PM my side project's app config and two weeks of uploads were gone.
Friday. Six PM. You know the feeling.
Nobody tells you this when you're getting into infrastructure security: the most dangerous attacker in your threat model has your SSH key, good intentions, and a deadline. It's you.
The old shape made me fragile
Everything lived on one VPS: app, database, cron, the nginx config only God understood. "One machine" means one blast radius — my typo didn't take out a service, it took out the estate. Recovery was a weekend of reconstructing configs from memory like an archaeologist.
I'd spent years defending that box from strangers and got taken out by myself in four seconds.
The new shape: small boxes, snapshot-first
I can't uninstall my own hands, so I moved to small single-purpose Cubes on Krova Cloud — Firecracker microVMs, own kernel each, no public IP — and made every risky change start the same way:
# before the scary thing
krova snapshots create web-1 --name pre-migration
krova snapshots list web-1
# the scary thing goes scary
krova snapshots restore web-1 snap_abc123 # disk rolls back; minutes, not weekends
Restore replaces the Cube's disk with the snapshot. A mistake now eats one small machine with one job; worst case I delete the Cube and redeploy. Note even the delete dialog defaults to "preserve a backup before deleting" — the platform assumes you'll hurt yourself eventually and pads the floor.
Backups that outlive the Cube
Snapshots are a seatbelt, not a vault. For the database Cube I keep Backups — they store the disk and the config (vCPU/RAM/disk/image/region/mappings) and survive deleting the Cube they came from. The workflow that saved my weekend:
- Redeploy the Backup into a fresh Cube. The original stays untouched — which makes this perfect for drills and staging copies.
- Verify what actually came back, because a green status is not your data:
ssh -p <port> ubuntu@<drill-cube-host>
systemctl is-active myapp # enabled units come back on their own
curl -s http://127.0.0.1:8080/ # the app answers
ls -la /srv/uploads # the files are actually there
- Tear the drill Cube down when done. One gotcha worth knowing: redeploying alongside a live original gives you a Cube with all the data and none of the domains (mapping conflicts are skipped, not stolen). Exactly right for a drill, exactly wrong if you expected a failover.
The monthly drill
Untested recovery is folklore. Mine runs while everything is calm:
#!/bin/bash
# restore-drill.sh — first Tuesday of the month
set -euo pipefail
# 1. redeploy latest Backup of db-1 into a fresh Cube (dashboard/API)
# 2. verify from outside my normal tooling
ssh -p "$DRILL_PORT" ubuntu@"$DRILL_HOST" '
systemctl is-active postgresql
pg_dump app_db | gzip | wc -c # sane, non-zero size
'
# 3. drill over
krova cubes delete "$DRILL_CUBE"
If the drill ever fails, it fails on a Tuesday, on purpose, with coffee — not at 6 PM on a Friday with adrenaline.
The honest part
Krova's own docs say it plainly, and I love them for it: "Backups are not a backup strategy on their own." Everything above lives on the same platform as the Cube. It covers bad deploys, deleted files, eager deletes — not the platform-shaped hole. So the database also dumps nightly to object storage I control, and the .cube download link (presigned, 15 minutes) gets treated like a credential, because it is one — anyone holding it holds your whole disk.
And small machines don't make you careful. I still typo. The point was never to become perfect. The point was to stop needing to be.
Design for the person with the key
Audit your worst incidents and you'll see the pattern: the expensive ones usually start with a person who had legitimate access and a bad moment. Small blast radii, snapshot-first habits, backups that outlive the machine, drills on a schedule. That's the whole architecture.
You'll deal with that person someday. I know because I've met them. It was me, on a Friday, at 6 PM.
