In my previous post, I wrote about a Linux routing issue that broke a deployment and caused repeated validation failures.
What ultimately led me to the root cause wasn't a configuration change or a log entry.
It was a packet capture.
This article isn't about a routing issue. It's about the tool that helped uncover it and the lesson I took away from the entire investigation.
Everything Looked Healthy
The first step was verifying the basics.
I checked whether the service was listening on the expected ports:
sudo ss -tulpn | grep -E ':80|:443'
Everything looked normal.
Next, I verified the application itself:
curl http://localhost
The application responded immediately.
I also verified DNS resolution and confirmed the domain was pointing to the correct public IP.
At this point, nothing looked obviously wrong. The application was healthy, the reverse proxy was healthy, and the network configuration appeared healthy. Yet validation attempts continued to fail.
The Logs Weren't Helping
The logs consistently showed variations of:
authorization failed
timeout during connect
likely firewall problem
The problem was that the logs only described the symptom.
They didn't explain why it was happening.
So I started investigating the usual suspects: DNS, firewall rules, reverse proxy configuration, and listening ports.
Everything continued to look fine.
The more I investigated, the less sense the issue made.
Looking Beyond The Logs
At some point I realized I was only looking at what the software was reporting.
I wasn't looking at what the network was actually doing.
So I decided to capture the traffic directly:
sudo tcpdump -ni ens3 tcp port 80
I triggered another validation attempt and watched the packets arrive.
Almost immediately, I saw something interesting:
IP 124.x.x.x > 51.x.x.x.80: Flags [S]
IP 124.x.x.x > 51.x.x.x.80: Flags [S]
IP 124.x.x.x > 51.x.x.x.80: Flags [S]
The requests were reaching the server.
That single observation completely changed the direction of the investigation.
The First Real Clue
Up until that moment, I had been operating under the assumption that external systems couldn't reach the server.
The packet capture proved otherwise.
Traffic was arriving.
The server was receiving connection attempts.
The problem wasn't inbound connectivity.
The problem was somewhere after that.
Instead of asking:
Why can't external systems reach my server?
I started asking:
If traffic is reaching the server, why isn't the connection completing?
That shift in thinking changed the entire investigation.
What tcpdump Revealed
To understand why the packet capture was so important, it helps to understand what a normal TCP connection looks like.
A healthy connection follows a three-way handshake:
Client -> SYN -> Server
Client <- SYN-ACK <- Server
Client -> ACK -> Server
What I was actually seeing looked more like this:
Client -> SYN -> Server
Client -> SYN -> Server (Retry)
Client -> SYN -> Server (Retry)
The incoming connection attempts were reaching the server, but the connection was never being established successfully.
That immediately ruled out several possibilities.
DNS wasn't the problem because requests were arriving.
The reverse proxy wasn't the problem because it was listening correctly.
The application wasn't the problem because it responded locally.
Within a few minutes, the packet capture had eliminated entire categories of potential root causes.
Following The Evidence
Once I knew inbound traffic was reaching the server, I shifted my attention toward the network path itself.
I started examining interfaces, routes, and outbound traffic behaviour using commands like:
ip route
and
ip route get 8.8.8.8
Those commands eventually exposed the real issue.
The server had multiple network interfaces, and outbound traffic was being routed through an unexpected path. That routing behaviour was causing validation attempts to fail even though the service itself was perfectly healthy.
The actual root cause turned out to be a Linux routing issue.
But I might never have found it if I hadn't first verified what was happening on the wire.
Why tcpdump Was The Turning Point
Before running tcpdump, I was relying entirely on logs and assumptions.
The logs suggested a firewall issue.
The packet capture showed requests reaching the server.
Those two observations pointed in completely different directions.
Without the packet capture, I probably would have continued tweaking firewall rules, reverse proxy settings, and application configuration.
Instead, the investigation moved toward routing almost immediately.
That's what made tcpdump so valuable.
It wasn't the tool that solved the problem.
It was the tool that revealed reality.
Lessons Learned
1. Logs Don't Tell The Entire Story
Logs are useful, but they're generated by software. They only describe what the application believes is happening.
Sometimes that's not enough.
2. Verify Assumptions Early
I spent time investigating DNS, firewall rules, and reverse proxy configuration because they seemed like the most likely causes.
The packet capture disproved those assumptions within minutes.
3. Packet Captures Can Change Everything
You don't need advanced networking knowledge to get value from tcpdump.
Even a simple capture can tell you whether traffic is arriving, leaving, or disappearing somewhere in between.
4. Eliminate Entire Categories Of Problems
One of the biggest advantages of packet captures is that they quickly rule things out.
Sometimes that's more valuable than finding the answer immediately.
Final Thoughts
This incident reminded me that assumptions can be surprisingly expensive.
The logs pointed toward a firewall issue. The services looked healthy. Everything seemed to suggest a particular problem.
But the moment I looked at the packets, the entire investigation changed direction.
Since then, whenever a network issue doesn't make sense, I try to reach for tcpdump much earlier.
Because logs tell you what software thinks happened.
Packets show you what actually happened.
