Disable AWS NLB cross-zone load balancing

This article explains why you shouldn’t enable cross-zone load balancing on your NLBs when using instance target groups with client IP preservation.

Types of load balancers

In the context of HTTP traffic:

Availability zones and load balancers

It’s best practice to run your applications in multiple availability zones. Load balancers allow you to distribute traffic over target instances hosted in multiple availability zones.

NLB cross-zone load balancing

Cross-zone load balancing means that each NLB endpoint can direct traffic to target instances in a different availability zone.

By default, it’s disabled:

NLB cross-zone load balancing disabled diagram

When enabled, it allows load balancing to target instances in different zones:

NLB cross-zone load balancing enabled diagram

Note that NLBs preserve the source IP and port of the ingress traffic.

The problem with NLB cross-zone load balancing

Suppose your TCP client is behind a NAT gateway of some sort, and is making plenty of requests to your service.

For each request:

  1. Resolve NLB CNAME to an IP address.
  2. Open a TCP connection to the destination of that NLB endpoint’s IP address.
  3. The NLB forwards TCP connections to a target instance. Since cross-zone load balancing is enabled, this target instance could be in zone A or B.

Everything could be fine, until:

NLB cross-zone load balancing TCP collision diagram