In September a developer asked for a copy of the orders database from the previous Friday to investigate a bug. The newest dump in the bucket was five weeks old. The backup panel on our main dashboard was green, as it had been every day of those five weeks.
The nightly dump used to run from cron on a database utility VM. At the end, the script wrote two lines to a file in the directory that node_exporter's textfile collector reads: backup_success 1 and the size of the dump. The collector reads every .prom file in that directory on each scrape and exposes whatever is in it. Our alert fired when backup_success was zero. The panel showed its latest value.
In August we moved the job to a Kubernetes CronJob, removed the cron entry from the VM and left the VM running for other tasks. The file stayed where it was, saying 1. node_exporter does not know how old the contents of a textfile are, and does not care. It serves the same numbers on every scrape for as long as the file exists, so Prometheus received a fresh sample of a five week old success every fifteen seconds.
The new CronJob, meanwhile, failed every night. Its service account had no permission to write to the bucket. It wrote no metrics at all, and an alert on a value cannot fire on a series that does not exist.
The scripts now publish a timestamp, backup_last_success_timestamp_seconds, and the alert fires when time minus that value is more than twenty six hours, so silence counts as failure. node_exporter also exposes node_textfile_mtime_seconds for every file it reads, and we alert on any file not modified for longer than its job's schedule. For CronJobs we alert on kube-state-metrics' kube_cronjob_status_last_successful_time, which belongs to the job and cannot be left behind on a VM. The old file is deleted. And once a week a job restores the latest dump into a scratch database and counts yesterday's orders, because a backup is finished when it has been read back.
The metric said a backup had succeeded. It was true once, and the exporter kept repeating it long after anyone was doing the work.
– Sergey Shinder
