Most teams never wonder what happens when a packet is sent to a Service IP — right up until first-packet latency starts behaving strangely. Until that moment kube-proxy is the quietest component in the cluster: nobody looks at its dashboard, nobody discusses its version, nobody asks "how is kube-proxy doing today?" Then the service count grows, first packets slow down, and somebody says: "let's drop kube-proxy and let Cilium handle it with eBPF."
The thesis of this piece is simple: removing kube-proxy is not a performance toggle, it is handing over the data plane. Moving service resolution from iptables chains to eBPF hash tables genuinely changes the scaling behaviour — but the same move makes you dependent on kernel versions, on cgroup layout, and on an entirely new debugging reflex. Both the gain and the cost are concrete, and I think deciding without putting them side by side is a mistake.
First, let's admit it: kube-proxy is still alive
When you hear "kube-proxy is dead," go read the docs. The Kubernetes virtual IPs and service proxies reference says that in 1.37 the default mode is still iptables, and that a future release will change the default to nftables. On Linux there are three modes — iptables, ipvs, nftables — and on Windows, kernelspace.
Keeping this picture current matters, because a lot of writing out there is two years behind. The nftables mode arrived as alpha in 1.29 and, according to the announcement on the Kubernetes blog, became stable in 1.33; that same post warned explicitly that even after GA, iptables would remain the default. Changing the default was left to a separate proposal, KEP-5343: alpha in 1.37, beta in 1.39, GA in 1.40, with users who rely on the implicit default warned via logs and events starting in 1.37. The nftables backend has its own kernel threshold too: 5.13 and later.
The practical conclusion is almost annoyingly plain for teams eyeing eBPF: the cheapest escape from the iptables mode's scaling problem is usually not removing kube-proxy, but changing its mode. Switching to nftables with one configuration line is a much smaller move than replacing the whole data plane.
But "smaller move" and "free move" are not the same thing. The same docs page lists several places where behaviour deliberately differs after the switch, and they are outage-class in production: NodePort services no longer listen on every local IP but default to --nodeport-addresses primary, meaning the node's primary IPv4 and/or IPv6 address; the "accept inbound traffic on this port" rules that iptables mode adds to survive overly aggressive firewalls are not added in nftables mode, so a local firewall is now your job; and the workaround iptables mode installs for the pre-6.1 kernel conntrack bug that closes long-lived TCP connections with "connection reset by peer" is not installed by default in nftables mode. Whether you depend on that last one is measurable — check kube-proxy's iptables_ct_state_invalid_dropped_packets_total metric, and if you do, --conntrack-tcp-be-liberal restores the behaviour.
Even so, that list is short next to the risk of replacing the data plane wholesale. Make the Cilium decision after you have measured the cheap step and found it insufficient.
The real issue is O(n) — and where it comes from
The problem with iptables mode is not that "iptables is slow"; it is that the rule count makes the work grow linearly. The Kubernetes docs note that a few rules are created for every Service and for each endpoint IP, that in clusters with tens of thousands of Pods and Services this becomes tens of thousands of rules, and that kube-proxy may take a long time to write those rules into the kernel. The nftables announcement makes a sharper point: because there is one rule testing each possible Service IP at the top level, matching the first packet of a new connection is O(n) in the number of Services. Both average and worst-case latency climb as the cluster grows.
Note what this is not: it is not a throughput story. Packets on an established connection do not re-traverse that chain, thanks to conntrack. What hurts you is the tax the first packet pays in workloads that create short-lived connections — a job opening a new connection per request, a health-check storm, a batch that scales out.
Cilium's kube-proxy replacement targets exactly this: it implements the same Service abstraction on top of eBPF hash tables, so lookup cost is independent of service and endpoint counts. And it adds one more thing: socket-LB. Cilium's kube-proxy-free documentation describes eBPF cgroup hooks attached to connect(), sendmsg() and recvmsg(). ClusterIP translation happens while the socket is being set up, before the packet ever reaches the network stack. The application connects to the service IP, the kernel connects the socket straight to the backend Pod IP, and there is no per-packet NAT.
Laid out this way, the cost becomes as visible as the gain: the translation no longer shows up in iptables-save output. Whoever goes looking for "where did this packet go?" will find their familiar tool returning nothing useful.
Kernel thresholds: an inventory task, not a romance
Cilium's system requirements page recommends kernel 5.10 or later for this version, with RHEL 8.10's 4.18 kernel accepted as equivalent. Its table asks for more for some features: 5.19 for IPv6 BIG TCP, 6.3 for IPv4 BIG TCP, 6.8 for netkit device mode. The kube-proxy-free page adds 5.16 for managed neighbor entries.
Socket-LB itself wants a cgroup v2 filesystem; Cilium mounts it at /run/cilium/cgroupv2 by default. There is also a frequently misread condition: the CONFIG_INET_DIAG, CONFIG_INET_UDP_DIAG and CONFIG_INET_DIAG_DESTROY kernel options are not there to make socket-LB work, but to allow sockets still attached to a deleted backend to be forcibly terminated. Without them socket-LB still runs; but after a backend is removed, Pods connected to it can keep sending traffic to a dead address for a while. The docs also enumerate patched kernel versions for setups that mount NFS/SMB over ClusterIPs: after 5.4.0-187, 5.15.0-113 and 6.5.0-41 on Ubuntu; after the specific z-stream kernels of RHEL 8.10 and 9.4 (4.18.0-553.8.1 and 5.14.0-427.31.1 respectively, or newer).
Dull to read, decisive in the decision. Two commands give you the inventory:
uname -r
stat -fc %T /sys/fs/cgroup
On my own VPS (no Kubernetes there — the blog infrastructure runs on plain containers) the output is:
6.8.0-138-generic
cgroup2fs
On a recent base like Ubuntu 24.04 you clear all of these thresholds comfortably. But if your fleet still has a long-lived 4.x node in the "don't touch it, it works" category, the story changes — and that node is usually the one carrying the most critical workload. My decision rule is blunt: moving to an eBPF data plane is first an OS inventory project; changing a Helm value comes afterwards.
The chicken-and-egg problem in the configuration
Cilium's Helm default is kubeProxyReplacement=false, and that does not mean "no eBPF at all": even then, per-packet in-cluster load balancing of ClusterIP services happens in eBPF. Setting it to true adds the rest — NodePort, hostPort and socket-LB. Three values turn it on:
cilium install \
--set kubeProxyReplacement=true \
--set k8sServiceHost=${API_SERVER_IP} \
--set k8sServicePort=${API_SERVER_PORT}
At first glance the last two lines look redundant. They are not. Without kube-proxy, the Cilium agent cannot reach the API server through the kubernetes.default ClusterIP — because the component that would resolve that ClusterIP is not up yet. A system trying to tie its own shoelaces while standing on them. So you hand it the API server's real address. This is the most common stumbling point in fresh installs, and because the error looks like "no networking," it pushes people in the wrong direction.
hostPort support is enabled automatically in this mode; in exchange, the docs are explicit that hostPort values must not overlap the NodePort range. Another detail is the NodePort range itself: if it overlaps the kernel's ephemeral port range (net.ipv4.ip_local_port_range), Cilium appends the range to the reserved ports. If you run a non-default NodePort range, declare the same range to Cilium via nodePort.range; otherwise you collide with local application traffic on the host. You can also tune load-balancing behaviour: loadBalancer.algorithm=maglev for consistent hashing, loadBalancer.mode=hybrid for a DSR/SNAT mix.
Migration: the one sentence that matters in an existing cluster
The documentation carries a clear warning about kube-proxy and the Cilium replacement running at the same time: the two mechanisms operate independently of each other, and existing connections will break during the transition. Coexistence is only reasonable on new clusters that are not yet serving user traffic.
On a new cluster the order is simple:
kubeadm init --skip-phases=addon/kube-proxy
On an existing cluster the cleanup has to be real, or leftover rules will come back to visit you months later:
kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy
# on every node, flush the remaining KUBE chains
iptables-save | grep -v KUBE | iptables-restore
kube-proxy also has its own cleanup flag; the command-line reference describes --cleanup as "cleanup iptables and ipvs rules and exit." If you are coming from IPVS mode, this route is tidier — you leave the virtual server entries to the component that created them instead of chasing them by hand. I collected the quirks specific to IPVS mode in an earlier post; knowing that baseline before the migration is what lets you tell what actually improved from what merely moved.
When you plan the maintenance window, accept this up front: it is not a "seamless live migration" scenario. If it were me, I would drain nodes one by one, convert node by node, and verify service reachability at every step.
Put the rollback in the same window, because it is not free either. The docs' warning is symmetric: connections should be expected to break both when the eBPF replacement is added to a running cluster and when it is removed and the work is handed back to kube-proxy. And if you performed the cleanup above, going back is not one command: alongside the DaemonSet you also have to restore the ConfigMap you deleted. Save both manifests before you enter the window.
Measure before you move: what exactly are you comparing?
The most frequently skipped step is recording the "before." You will not be able to prove "it got faster" to anyone six weeks later, because three other things will have changed in the same week.
The quantity worth comparing is not throughput but the time the first packet of a new connection pays. A practical method: from a client Pod, open thousands of sequential short-lived connections to the target ClusterIP and chart the distribution of connect() durations — write down p95 and p99, not the average. Take the same measurement under two loads: with few Services in the cluster and at your real production service count. If the iptables-mode curve bends upward in the second measurement, you have observed the O(n) behaviour the docs describe in your own cluster; if it does not bend, your problem is somewhere else and swapping the data plane will not fix it.
The second record is rule-programming time. In iptables mode kube-proxy rewrites the rule set into the kernel as endpoints change; this is where the docs say updates "may take a long time." If you do not know where that duration climbs during busy deploy hours, you will have no yardstick for saying "it is more stable now" afterwards. kube-proxy exposes its own metrics endpoint; record the two weeks before the migration and that is enough.
The third is conntrack. In iptables mode every new connection creates a connection-tracking entry; once socket-LB is in play, translation for service traffic leaving a Pod happens at the socket layer, so the table behaves differently. If you do not note the occupancy trend beforehand, you will have no "it used to sit here" number to point at later.
Source IP does not change, it just changes clothes
externalTrafficPolicy behaves the way you already know: the Cluster default applies SNAT and the client source IP is lost, while Local preserves the source IP and confines external traffic to backends on the node. There are two differences here. First, Cilium offers a third option: in DSR mode the source IP is preserved, and policy on the backend node can match on it. Second, a less known relaxation: the docs say in-cluster connectivity to Local services also works from nodes that have no local backends, because SNAT is not needed there, so all endpoints stay available for load balancing on the in-cluster side.
That detail has direct consequences for teams writing network policy. If you have a rule restricting external traffic by source IP, which mode you run changes what the rule means. I have written before about the discipline of tightening policies gradually; the "observe first, enforce later" approach there applies here too — especially in the week you change the data plane.
Blind spots: where socket-LB does not look
The most valuable information is buried in the limitations section. Because socket-LB translates inside the Pod's namespace at the socket layer, some workloads either do not benefit from it or actively suffer: the documented examples are redirection scenarios that rely on the original ClusterIP within the pod namespace (Istio sidecar) and Pods whose nature puts them outside that socket path (KubeVirt, Kata Containers, gVisor). For these there is socketLB.hostNamespaceOnly=true, and the name can mislead: what it does is bypass socket-LB in the pod namespace and fall back to the tc load balancer at the veth interface. It is not a fix but a deliberate retreat — your ordinary Pods also drop out of socket-layer translation.
For teams running VMs on Kubernetes this is not a small footnote, it is an architectural decision. If virtualization on the cluster is on your roadmap, read that line before the migration.
The same setting shows up in a second, sneakier place: the docs note that when the maps used for socket termination fill up, LRU eviction can drop entries belonging to active connections, and from that point socket termination can stay unreliable until the node is restarted. In clusters with long-lived, heavy UDP traffic, that line deserves a deliberate read.
The short list of remaining limits: SCTP is not fully supported, with TCP and UDP being the properly covered protocols. DSR NodePort mode is incompatible with TCP Fast Open; if you want both, the docs tell you to fall back to SNAT. And on the masquerading side, the docs recommend BPF masquerading over iptables to reduce the risk of port collisions.
There is an architectural consequence too: once kube-proxy is gone, the only component implementing Services is the Cilium agent. Its upgrades and its failures are now directly a service-reachability matter.
Maglev, DSR and XDP: how far should you go?
Once the replacement is on, a menu appears, and the urge to enable everything at once is strong. I would argue the opposite: stability with the plain setup first, then one at a time.
loadBalancer.algorithm=maglev enables consistent hashing. The benefit is that when the backend set changes, most existing flows keep landing on the same backend — meaningful for long-lived connections and for setups that want consistency across nodes. The default random distribution is fine for most workloads, so unless you have a symptom telling you otherwise, leave it alone.
DSR (direct server return) sends response traffic back to the client while skipping the intermediate node, and preserves the source IP as a bonus. The cost is a set of assumptions about the network. What matters here is that DSR is not a single implementation: loadBalancer.dsrDispatch offers three. The default, opt, carries the information in an IPv4 option or IPv6 extension header, and some fabrics drop those packets. geneve works in both native routing and tunnelling mode. ipip avoids the Cilium-specific IP options, which the docs call a robust choice in environments where the other dispatch methods hit connectivity issues; in return it only works with native routing and requires the frontend port to equal the backend port. All three share one limit: no TCP Fast Open. For a mixed setup there is loadBalancer.mode=hybrid: some traffic over DSR, the rest over SNAT.
XDP acceleration is the far end of the tail. It depends on native XDP support in the NIC driver; in the docs' own wording, the majority of drivers supporting 10G or higher rates also support native XDP on a recent kernel. If you are not pushing serious NodePort traffic through an edge node, you may never reach this step, and never reaching it is not a bad outcome.
The general principle: these three look like a performance menu, but each adds an assumption about the fabric, the NIC or the protocol. Every new assumption is one more item on your checklist when something goes wrong.
Verification: don't guess what is enabled
After the migration, these three outputs are enough:
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose
bpftool cgroup tree /run/cilium/cgroupv2/
The verbose output shows which sub-feature (NodePort, hostPort, socket-LB, XDP acceleration) is running in which mode. Build this habit from day one: in an eBPF data plane, "I think it's on" costs more than it did in the iptables world. There, at worst, you were reading a rule listing; here you have to ask the state directly.
A decision framework
One note on version ground, because this area ages fast: according to Cilium's releases page, 1.20.0 shipped on 29 July 2026 and 1.20.1 is dated 18 August 2026. The behaviours above come from that stable release's documentation; defaults can shift in the next minor, so open your own version's page before migrating.
Roughly, I would draw the lines like this:
-
Try the cheap route first. If your problem is first-packet latency and your kernels are 5.13+, moving kube-proxy to
nftablesmode is a far smaller step. Read up on the NodePort address default, the firewall rules and the conntrack workaround, then measure; if it is enough, stop there. - Take the kernel inventory. If the fleet has nodes below 5.10 (RHEL 8.10 excepted), this is not a networking project, it is an OS upgrade project.
-
Count your sidecar and VM workloads. If you use Istio sidecars, KubeVirt, Kata or gVisor, plan for the socket-LB blind spot and the cost of the
socketLB.hostNamespaceOnlybypass from the start. - Plan both the migration and the rollback as outages. Connections break both when you add the replacement and when you remove it; do not begin without backing up the kube-proxy manifests.
-
Update the debugging reflex.
cilium-dbg statusandbpftoolinstead ofiptables-save; everyone on call needs to know that, not just the person who ran the migration.
Let me restate the thesis, because the decision lives there: removing kube-proxy means trading performance for observability and dependency surface. The trade is not a bad one; O(1) lookup in a cluster with tens of thousands of services is a genuine gain, and the simplicity socket-LB brings is pleasant. But you should make the trade knowingly. Black boxes spend their worst nights without telling anyone; get to know your new one before you need it.
