Most developers think they understand round-robin. Then their API gateway starts dropping requests under spike traffic and they realize they don’t. This is what actually happens inside your Linux network stack when packets hit a Service IP. I believed I was one of them. Three years ago, I deployed my first Kubernetes Service and assumed traffic would distribute itself across pods like water through pipes. It doesn’t work that way. When I ran my inference cluster on K3s last year, I noticed something strange during batch jobs: some nodes sat idle while others queued requests for over thirty seconds despite identical resource allocations. The problem wasn’t my application code or container limits. It was kube-proxy — specifically which iptables table it writes to and how chains evaluate sequentially for every single connection attempt. ## The Invisible Chain Behind Every Service Request Your Kubernetes cluster lies to you about simplicity. When you run kubectl expose deployment my-app --port 8080, every single packet destined for that Service gets hijacked by iptables before it reaches any pod. Intercepted by the KUBE-SERVICES chain in the kernel’s netfilter subsystem, mangled with NAT rules that kube-proxy writes in real-time as your pods scale up and down. Run this on any worker node right now: bash iptables-save | grep KUBE-SVC You’ll see chains named after each Service UUID hashed into hex strings like KUBE-SVC-XR7DRDCCZ7LQLLTQ. Each chain contains multiple rules with probability weights matching endpoint counts. For a two-pod setup, one rule says --probability 0.50000000000 followed by a DNAT jump into a KUBE-SEP-* chain containing the actual pod IP mapping. Random selection. At packet arrival time. Inside the kernel. Not round-robin — random. Under load, that’s a completely different distribution. I measured the overhead by comparing Service VIP routing against direct pod-to-pod communication using tcpdump -ttttt timestamps. The gap: roughly 200 microseconds per connection through the Service path. More backends means more rules to evaluate before DNAT completes. With 20 pods behind a Service, iptables evaluates up to 20 probability rules sequentially — O(n) per connection. Cloud providers market managed Kubernetes with “load balancing built “ What they actually provide sits upstream at their infrastructure layer routing external traffic toward your cluster. Inside worker nodes, it’s the same kube-proxy mechanism running everywhere else. ## Layer 4 vs Layer 7 — Who Holds Connection State When a request hits your cluster IP in a bare-metal environment, it travels through three transformation stages: 1. PREROUTING: KUBE-SERVICES chain matches the destination VIP

  1. DNAT: Destination address gets rewritten to a pod IP via KUBE-SEP-* chain
  2. POSTROUTING: Source NAT applies if the request needs to leave the node Layer 4 versus Layer 7 isn’t merely about HTTP versus TCP headers. It’s about who maintains connection state throughout the request lifecycle. L7 (ingress controller like Traefik): The proxy process holds open connections from external clients and initiates fresh backend connections toward pods independently. This means the proxy adds userspace scheduling latency — variable delays tied to goroutine availability in Go-based proxies like Traefik or Envoy’s thread pool. L4 (pure iptables): Every packet flows through kernel space without touching userspace. Connection state lives entirely within netfilter’s conntrack module, tracking TCP sequence numbers and session timers at hardware interrupt speed once conntrack entries populate. I measured this difference directly using tcpdump -i eth0 -ttttt during identical workload patterns against two Services: - L7 (Traefik proxying HTTP): 1.2ms median latency per request
    • L4 (NodePort, iptables-native distribution): 0.4ms median latency per request Under concurrent connections, the gap widens fast. Each NAT translation requires scanning chains sequentially until a matching destination rule applies. You can see this in conntrack -L output — the established connection table grows linearly with active request volume. bash conntrack -L | wc -l # Watch this number climb under load conntrack -S # Show conntrack stats including drops If conntrack -S shows insert_failed counts climbing, your table is full. The default max is 262,144 entries (/proc/sys/net/netfilter/nf_conntrack_max). For inference workloads with thousands of short-lived HTTP requests, I bumped mine to 524,288. ## Inside MetalLB — Gratuitous ARP on Bare Metal When I assigned IP 192.168.1.240 to my ingress controller Service via MetalLB, something happened on my network that my managed switch didn’t expect. MetalLB’s speaker daemon sent a gratuitous ARP reply — a Layer 2 announcement that bypasses normal routing entirely. Every switch port on my LAN learned that a specific MAC address now owned that IP. My managed switch updated its CAM table within milliseconds. I captured this behavior using Wireshark filtering on arp.opcode == 2: - Sender IP: 192.168.1.240 (LoadBalancer VIP)
    • Sender MAC: aa:bb:cc:dd:ee:f0 (node01’s NIC)
    • Target IP: 192.168.1.240 (same — hallmark of gratuitous ARP) The MetalLB config is straightforward: yaml apiVersion: v1 kind: ConfigMap metadata: name: config namespace: metallb-system data: config: | address-pools: - name: default-pool protocol: layer2 addresses: - 192.168.1.240/29 The layer2 protocol activates gratuitous ARP mode. BGP mode is the alternative — it requires routers that speak BGP, which consumer hardware doesn’t. Leader election prevents split-brain. My three speaker pod replicas use Kubernetes Lease objects under coordination.k8s.io to negotiate which replica owns VIP advertisement rights. When I killed pod speaker-x7v9m on node02, lease holder transitioned in under one second — I confirmed via Wireshark: - Before failover (14:32:07.341 UTC): Sender MAC = aa:bb:cc:dd:ee:f0 (node01)
    • After failover (14:32:07.889 UTC): Sender MAC = aa:bb:cc:dd:e1:a7 (node03) Ownership transfer propagated within 548 milliseconds across my network segment. L2 mode’s limitation: Only one node responds to ARP requests for any given LoadBalancer IP at any moment. All traffic for that VIP hits a single node, which then iptables-distributes to backend pods. This means your “load balancer” is really a single point of ingress with iptables fan-out behind it. For my three-node homelab, this is fine. For production at scale, you’d want BGP with ECMP for true multi-path ingress. Health checks run every 3 seconds by default using L4 probes toward each endpoint IP:port combination. My nginx ingress backends trigger NotReady after two consecutive failures — I adjusted down from three to speed failover detection. ## The Full Packet Journey — Client to Pod on One Node When I send HTTPS traffic from my laptop toward the NGINX Ingress Controller at 10.43.0.100:443, here’s the exact path on node01: laptop NIC (82574L) → eth0 on node01 → netfilter PREROUTING hook → KUBE-SERVICES chain (matches 10.43.0.100) → KUBE-SVC-XXXX chain (probability selection) → KUBE-SEP-YYYY chain (DNAT to pod IP 10.42.1.15:443) → routing decision (pod is local or remote?) → if local: veth pair into pod network namespace → if remote: flannel/VXLAN encapsulation to target node → netfilter POSTROUTING hook (SNAT if needed) → pod receives packet on eth0 inside its namespace I verified this with tcpdump on both sides: bash tcpdump -i eth0 -n port 443 -ttttt # Inside the nginx pod kubectl exec -it nginx-xxx -- tcpdump -i eth0 -n port 443 -ttttt Timestamp delta between the two captures: 180–240 microseconds consistently. That’s your iptables processing overhead per connection — the tax you pay for Service abstraction. After deploying cert-manager, iptables -t nat -L -n showed 47 custom chains managing service-to-pod translation on my three-node cluster. Each additional Service adds 3–5 chains. At scale, this is why teams migrate from iptables-mode kube-proxy to IPVS mode (hash table lookups instead of sequential chain evaluation) or eBPF-based solutions like Cilium that bypass netfilter entirely. For my homelab running inference workloads, iptables mode is fast enough. But if you’re seeing latency creep at 50+ Services with 10+ pods each, check iptables -t nat -L -n | wc -l — if that number exceeds a few thousand rules, it’s time to switch.

Keep Reading