The line that started this was on an invoice. Inter availability zone data transfer, six thousand three hundred pounds for the month, up from about nine hundred a year earlier, against a cluster whose external traffic bill had barely moved.
We run three zones because that is what you do. Sixty services talk to each other through ordinary cluster services, and a cluster service picks an endpoint from the whole set without any interest in where the caller is standing. With replicas spread evenly across three zones, a call stays local one time in three. Our checkout path makes fourteen internal calls before it answers, so in practice almost every request we serve crosses a zone boundary several times, is charged for leaving one and arriving at the other, and pays about a millisecond each way for the privilege.
Once we measured it rather than assumed it, the second number was the more interesting one. Eleven milliseconds of a hundred and eighty millisecond budget was distance. Forty one terabytes a month of our traffic never left the cluster in any sense a person would recognise and was billed as though it had crossed the country.
We turned on topology aware routing for the twelve chattiest services, which keeps traffic in the caller's zone while there are enough healthy endpoints there and spills over to the whole cluster when there are not. That second half is the part that needs checking before you enable it, because a service with three replicas across three zones has exactly one endpoint per zone and one restart away from falling back anyway. We went from four replicas to six on those twelve, which cost less than the transfer it saved. Two services stayed deliberately zone blind, because they sit in front of the payment gateway and I would rather pay for the hop than concentrate their traffic.
The bill came down by about seventy percent and the checkout p99 by nine milliseconds, from a change that touched no application code.
A cluster network is presented to your code as flat, uniform and free. It is none of those things. Distance survives every abstraction you put on top of it, and it turns up as latency and as a line on a document most engineers never see.
– Sergey Shinder
