I had an EC2 instance running an application.
Everything looked fine.
Then I changed one Security Group rule.
Suddenly…
the application couldn't reach the database.
The instances were healthy.
The database was healthy.
CPU was normal.
No deployment happened.
Just… timeout.
And that's when I learned something production-grade:
A Security Group is stateful, but that doesn't mean your architecture is automatically stateful.
The database allowed inbound traffic from the application's Security Group.
So when the app initiated:
EC2 → DB : 3306
the return traffic didn't need a separate inbound rule on the application side.
But when we accidentally restricted the egress from the application SG…
💀 the connection died.
Nothing was "down."
The network policy was.
That's the scary part about cloud infrastructure:
Your application can be perfectly healthy and still be completely unreachable because the control plane says "no."
And that's why production debugging isn't just:
“Is the server running?”
It's:
“Who is allowed to talk to whom, on which port, in which direction, and through which path?”
