You spend time optimizing a function to be 40 milliseconds faster, but then realize your cloud bill is so high that it's eating up all your savings.
I wish someone had told me over a cup of coffee, "you lose money on the network." (Here's the coffee chat I wish I'd had.)
The $30k hallway tax
I heard about a startup that received a warning. They were about 48 hours away from a $30,000 AWS invoice.
It's not from a computer. Not from a GPU farm running all night.
From a worker fetching millions of small JSON files out of S3. Innocent. Boring. Bankrupting.
It wasn't S3 that caused the issue. The real culprit was the NAT gateway that acted as a mediator between their private subnet and the internet. Each byte accessed by the worker had to go through this gateway, which charges based on the amount of data used.
Accessing S3 from within a VPC gives the illusion of locality. But it's not the case unless you explicitly make it so. Send the traffic through a NAT gateway and Amazon counts the packets to the bit.
Why juniors never see it coming
Computational cost can be expensive. When you start using a high-performance machine, you immediately notice the cost involved.
You only pay a tiny amount for each gigabyte transferred, but when you're dealing with millions of small files, these costs can quickly add up.
Here's the trap nobody flags:
→ Small files are worse than big ones. Overhead per request stacks up.
→ NAT gateway pricing is per-GB processed, on top of the per-hour charge.
→ Your code looks fine. The architecture is the bug.
→ The bill lands weeks later, long after the deploy that caused it.
The real kicker is that you send it on a Monday, you get billed at the end of the month, and by that time, you have already forgotten which loop you sent.
The fix is embarrassingly cheap
The startup didn't rewrite their application. They simply introduced a VPC endpoint for S3.
A gateway VPC endpoint allows connectivity from within your VPC to the S3 service. This means it enables you to route traffic to S3 through the VPC's private subnets rather than routing the traffic through the internet. This also means you can use VPC security controls and configurations that you've applied within the VPC to secure the traffic bound for S3.
That was the only routing configuration that prevented a bill in the five figures.
The lesson generalizes. Before you scale any worker that moves data, ask a dumb question:
→ Where does this traffic physically go?
→ Does it cross a NAT gateway, a region boundary, or the open internet?
→ Am I paying rent for every hop?
Traffic within the same region and the same availability zone (AZ), as well as traffic that is endpoint-routed, are relatively inexpensive or even free. On the other hand, traffic that goes across AZs, regions, or is NAT (Network Address Translation) routed, can lead to costs that slowly drain a startup's budget.
Network cost is a performance axis
Performance is how fast something runs and how much work it's able to do. When it comes to computers or networks, performance is measured by the time it takes to complete tasks (latency) and the amount of work that can be done in a certain amount of time (throughput). The faster a computer system responds to a user's actions or a network delivers data, the better its performance.
And here's a third aspect that often gets overlooked: the cost per operation.
An application that needs to access a million files in object storage "quickly" can put a huge strain on a network or system if not carefully engineered. A great chunk of the benefit of object storage is lost if network latency means a connection needs to be re-established for each file.
I've gotten so used to it that every arrow between two boxes is a question: who foots the bill for this wire, and what's the cost per byte?
Writing quick and dirty code for a poorly managed company is just a more sophisticated way of failing. 🙃
The takeaway
The cloud is marketed as if information can magically move from place to place. But reality is, it moves slowly, and you have to pay for every bit of transfer, and the cost accumulates regardless of your awareness.
Understand the journey of your bytes before you resize the component transporting them. A VPC endpoint is cheaper than a lunch. But a NAT gateway will blow your budget.
I have a question for you: What is the stupidest cloud bill you ever paid, and what was it for?
