In May our checkout returned 503 for eleven minutes, starting at ten past three in the morning. By the time anyone looked it had recovered on its own. No pod had restarted, none had been killed for memory, and the application logs said nothing. The ingress controller's logs said the checkout service had no endpoints.
The obvious next step was kubectl get events, and there were none for the night.
Kubernetes events are objects in the API like any other, and the API server deletes them after a fixed time, set by its event-ttl flag. The default is one hour. On a managed control plane, as ours is, you cannot change it. Much of what a cluster decides exists only as events: a readiness probe failing, a pod being evicted, a pod that cannot be scheduled. The pod's own status keeps the last time each container terminated, but a readiness failure terminates nothing, so it leaves no mark there at all. By four o'clock the cluster had forgotten.
We spent two days on theories about the database and the network. In the meantime we deployed an event exporter that writes every event to our log store as it happens. A week later it happened again, at five past three, and this time we could read it: Readiness probe failed, context deadline exceeded, on every checkout pod for nine minutes. A nightly reindexing job with no CPU limit had been scheduled onto the same nodes. The checkout pods were alive and answering, in about a second and a half, and their probes had the default timeout of one second. Every pod went unready at once, and the service had nothing to send traffic to.
The batch job now runs on its own tainted node pool. Checkout's readiness probe has a three second timeout. Events are kept for thirty days with our logs, warning events are counted by reason and namespace as a metric, and an alert fires when a service has fewer than half of its endpoints ready, whatever the pods themselves report.
A cluster writes down everything it decides and keeps the note for an hour. Anything you will want to read in the morning has to be copied somewhere else before then.
– Sergey Shinder
