In July, during an incident on our order service, I searched for its logs from the previous three days and found nothing after a Tuesday afternoon. It was not only that service. Every pod on every node had stopped reaching our log store at the same moment, and nobody had noticed for three days.
On that Tuesday we changed the log shipper DaemonSet to run with hostNetwork set to true. Our node monitoring scrapes agents on the node's own address, and that was the simplest way to give the shipper's metrics endpoint one. The pods restarted cleanly, the metrics appeared, and the change was closed.
The shipper sends to our log gateway by its service name in cluster.local. A pod's DNS policy defaults to ClusterFirst, which sends queries to the cluster DNS. The Kubernetes documentation has a sentence about hostNetwork that none of us had read: a pod on the host network with ClusterFirst falls back to the Default policy, which means it uses the node's own resolver configuration. Our nodes resolve through the cloud provider's DNS, which has never heard of cluster.local. Every connection attempt failed to resolve the gateway, the shipper retried and buffered, its buffer filled, and on each node the kubelet went on rotating container logs at ten megabytes, deleting the files the shipper had not read.
The pods stayed Running and Ready the whole time. Their readiness check asked whether the process was up, and it was.
We set dnsPolicy to ClusterFirstWithHostNet, which is the policy meant for exactly this case, and logs flowed again within a minute. The three days on the nodes were gone. An admission policy now rejects any pod with hostNetwork true unless it sets that policy, or says explicitly that it needs the node's resolver. The shipper's readiness check fails when its output has errored for two minutes. And we alert on log volume received per node: a node that sends nothing for ten minutes is either empty or blind, and we would rather be told which.
A switch named network also changed where the pod looks up names. We read it as a change to one port, and it was a change to everything the pod could find.
– Sergey Shinder
