Inside Kubernetes Service Networking: Follow a Packet from ClusterIP to Pod

You already know that a Kubernetes Service provides a stable IP address for accessing Pods.

But what happens inside the cluster when you run:

curl http://web-service

How does the request reach a backend Pod? Who redirects the packet? And what happens when that Pod is replaced?

Let's follow a packet from the Service's ClusterIP to a backend Pod and explore DNAT, connection tracking, and Linux networking rules.

1. Start with a Simple Service

Consider this Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

Assume Kubernetes assigns the Service this ClusterIP:

10.96.100.10

The Service accepts traffic on port 80 and forwards it to eligible backend Pods on port 8080.

Suppose one backend Pod has this IP address:

10.244.2.8:8080

The client sends a request to 10.96.100.10:80. The networking datapath must direct that traffic to an appropriate backend.

2. What Happens Behind the Scenes?

Several components work together:

• Service: Provides a stable virtual IP and port.
• EndpointSlice: Tracks the backend endpoints associated with the Service.
• EndpointSlice controller: Maintains EndpointSlices for selector-based Services.
• kube-proxy: Watches Service and EndpointSlice changes and programs forwarding rules when kube-proxy is used.
• Linux kernel: Processes packets using the configured networking rules.
• CNI networking: Provides connectivity between Pods, including communication across nodes when necessary.

In a typical kube-proxy iptables setup, no dedicated userspace process listens on the ClusterIP. Instead, the Linux kernel applies networking rules to redirect traffic.

Other implementations, including eBPF-based datapaths, can handle Service forwarding differently.

3. Follow the Packet: From ClusterIP to Pod

Imagine the client Pod runs on Node A, while the backend Pod runs on Node B.

Client Pod — Node A
10.244.1.10:50000
        |
        | Destination: 10.96.100.10:80
        v
Service ClusterIP
10.96.100.10:80
        |
        | DNAT
        v
Backend Pod — Node B
10.244.2.8:8080
        |
        v
Application processes the request

Let's examine the packet's journey.

Step 1: The client sends a request

The client application opens a TCP connection to:

10.96.100.10:80

This is the Service address, not the backend Pod's IP address.

Step 2: Networking rules select a backend

In a typical kube-proxy iptables configuration, kube-proxy programs rules that direct Service traffic through chains associated with the Service and its endpoints.

A simplified representation is:

PREROUTING
    |
    v
KUBE-SERVICES
    |
    v
Service-specific chain
    |
    v
Endpoint-specific chain
    |
    v
DNAT to 10.244.2.8:8080

These chain names illustrate the general flow; actual rules and packet paths depend on the implementation and configuration.

The important distinction: kube-proxy programs and updates the rules, while the Linux kernel processes packets against them. kube-proxy does not need to inspect every packet in userspace.

Step 3: DNAT changes the destination

DNAT stands for Destination Network Address Translation.

Before translation:

Destination: 10.96.100.10:80

After translation:

Destination: 10.244.2.8:8080

The destination changes to the selected backend Pod's IP and target port.

The application still connects to the Service address. It does not need to discover or track individual backend Pod IPs.

Step 4: The packet reaches the backend Pod

Because the backend Pod is on Node B, the cluster's networking implementation must deliver the packet to that node.

Depending on the CNI and cluster configuration, this can involve ordinary routing, packet encapsulation, or other datapath mechanisms.

The backend Pod receives the request on port 8080, where its application is listening.

This is where Service forwarding and the underlying Pod network work together.

4. What Happens to the Response?

The backend application sends a response to the client.

In a typical connection-tracked NAT setup, Linux connection tracking records the connection's translation state. This allows the networking stack to apply the corresponding reverse translation to response packets.

From the client's perspective, the response belongs to the connection it opened to the Service address.

Source NAT (SNAT) may also be used on some traffic paths, depending on the networking configuration. It can change the source address seen by the backend, so whether the original client IP is preserved depends on the path and implementation.

5. What Happens When a Pod Is Replaced?

Suppose a backend Pod is deleted or becomes ineligible to receive traffic.

An EndpointSlice might initially contain:

10.244.2.8
10.244.2.9
10.244.3.5

After an endpoint change, the eligible set might become:

10.244.2.9
10.244.3.5
10.244.3.12

The replacement Pod has a different IP, but the client can continue using the same Service address:

10.96.100.10:80

In a kube-proxy-based cluster, kube-proxy observes the updated Service and EndpointSlice information and reconciles its forwarding rules. Other datapaths have their own mechanisms for updating endpoint selection.

There is an important limitation: new connections can use the updated set of eligible backends, but existing connections may fail if their selected backend disappears. A stable Service IP does not guarantee that every connection survives a Pod replacement.

6. Does Every Cluster Use iptables?

No. Kubernetes Service forwarding depends on the proxy implementation and cluster configuration.

The nftables mode became stable in Kubernetes 1.33. It is the recommended kube-proxy replacement for Linux nodes where the kernel and networking components support it, but iptables remains the default for compatibility.

IPVS is deprecated in Kubernetes 1.35. The current roadmap expects it to be disabled by default in Kubernetes 1.40 and removed in 1.43; these are planned milestones, not completed removals.

Why does this matter? A troubleshooting command that works for an iptables-based cluster may not reveal the relevant rules in a cluster using nftables or an eBPF-based datapath.

7. Troubleshoot Service Traffic Step by Step

If a Service cannot reach its backend, investigate each layer instead of immediately blaming kube-proxy.

Step 1: Check the Service

kubectl get svc web-service
kubectl describe svc web-service

Verify the ClusterIP, Service port, target port, and selector.

Step 2: Check the Pods and labels

kubectl get pods -o wide --show-labels

Confirm that the backend Pods have the expected labels and are running.

Step 3: Check EndpointSlices

kubectl get endpointslices \
  -l kubernetes.io/service-name=web-service

If there are no eligible endpoints, investigate the Service selector, Pod labels, readiness, and endpoint conditions.

Step 4: Test the backend directly

From a location that can reach the Pod IP, try:

curl http://10.244.2.8:8080

Use an actual backend Pod IP and the port on which the application listens.

If this fails, investigate the application, Pod, listening port, and underlying network first.

Step 5: Test through the Service

curl http://10.96.100.10:80

If the direct Pod test succeeds but the Service test fails, investigate Service configuration and the active networking datapath.

Step 6: Inspect the networking rules

For a cluster using iptables:

sudo iptables-save -t nat

For a cluster using nftables:

sudo nft list ruleset

These commands may require node access and elevated privileges. Their output depends on the active implementation.

For deeper troubleshooting, tools such as tcpdump can help identify where packets stop. For example:

sudo tcpdump -ni any host 10.96.100.10

Packet captures may require access to the node, and traffic may not appear at every interface or in the same form after address translation. With eBPF-based networking, use the observability tools provided by that implementation.

8. Key Takeaways

A Kubernetes Service provides a stable destination while its backend endpoints can change.

In a typical kube-proxy iptables setup:

1. The client sends traffic to the ClusterIP.
2. Linux networking rules select a backend.
3. DNAT changes the destination to the Pod IP and target port.
4. The cluster network delivers the packet to the backend.
5. Connection tracking helps handle the response path.
6. Endpoint changes update the forwarding configuration for subsequent traffic.

Understanding this flow helps you troubleshoot networking layer by layer: Service configuration, EndpointSlices, packet translation, and Pod connectivity.

Instead of treating Kubernetes networking as a black box, you can follow the packet and investigate where the problem occurs.

Related reading: This article builds on my earlier guide, How Does Traffic Actually Reach Your Pods? Kubernetes Services & kube-proxy Explained, which covers Service types, selectors, EndpointSlices, and kube-proxy fundamentals.

Next in the series: Kubernetes Ingress Explained — How External HTTP Traffic Reaches Your Applications.