Understanding Load Balancing Costs in Multi-AZ Kubernetes Environments

The Burden of Default Load Balancing

If you've deployed a multi-AZ Kubernetes cluster paired with a service mesh, chances are you could be incurring costs that aren’t immediately apparent on your monitoring dashboards. Many organizations configure their systems relying on defaults that may not serve them well when operating across multiple availability zones (AZs).

How Traffic is Distributed by Default

Kubernetes Services alongside Istio's Envoy sidecars typically distribute traffic randomly across all active endpoints, without factoring in the specific AZ locations of the pods. This might be acceptable in a deployment concentrated in a single AZ, but as soon as workloads span across three or more AZs, that same distribution strategy can lead to significant inefficiencies. For instance, a pod in us-east-1a may have a 66% chance of connecting to a service in a different AZ. When a single user request traverses through several services and connects with database replicas, it can result in multiple cross-AZ interactions, adding to the overall latency.

The Dual Cost Impact

This default distribution pattern has tangible downsides, namely:

• Latency: For a typical production cluster handling around 3,500 requests per second (RPS) across three AZs, tests showed that requests localized to the same AZ had a p50 latency of 15-18ms, while those crossing AZs jumped to 25-30ms. Over time, with approximately two-thirds of requests crossing zones, the weighted p50 latency averaged around 24ms, compared to the ideal of about 17ms. This latency can compound rapidly, especially with multiple services involved in request processing.
• Monetary Costs: AWS charges $0.01 per GB for data transfer across AZs, which might seem negligible per request. However, internal service-to-service communications often generate five to ten times more traffic than external requests due to frequent API calls, replica reads, and data streaming. In the analyzed environment, cross-AZ data transfer costs conservatively reached approximately $600 each month, not accounting for additional expenses such as automatic cross-AZ replication.

Navigating Load Balancing Solutions

Istio offers a locality-aware load balancing feature via the DestinationRule traffic policy. The first instinct might be to direct all traffic to local AZs, but this strategy can severely limit your resilience by exposing your system to risks during malfunctions or spikes.

A more prudent approach is to implement a weighted traffic distribution — for example, routing 80% of the traffic to the local AZ while distributing 10% to each of the other two. This, paired with outlier detection, allows for automatic exclusion of unhealthy endpoints, which helps maintain service performance under increased load.

Requirements for Effective Implementation

To successfully apply locality-aware traffic management, here are three prerequisites that demand attention:

1. Balanced Pod Distribution: Maintain even distribution of pods across AZs. If one AZ hosts 50% of the service’s pods, an 80/10/10 routing policy will skew utilization heavily toward that AZ. Utilizing topologySpreadConstraints with maxSkew: 1 helps ensure equitable distribution.
2. Ingress Gateway Pods Diversity: Locality-aware routing will fall short if ingress gateway pods are concentrated in one AZ. Strategically spreading them across all AZs is vital.
3. Disable NLB Cross-Zone Load Balancing: Ensure the annotation service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false" is applied. Without this, the load balancer might introduce unintentional randomization that contradicts your efforts at the mesh layer.

Benefits and Tradeoffs

Implementing locality-aware load balancing can lead to reduced latency and enhanced system resilience. In scenarios where an AZ fails, the previous load distribution model would cause the remaining AZs to face a sudden spike in demand, risking overload and cascading failures. In contrast, with an 80/10/10 distribution, the remaining zones only need to absorb a smaller fraction of the overall load, inherently managing stress better.

However, this enhanced performance comes with increased operational complexity. The traffic distribution isn't intuitively balanced like traditional methods, leading to potential confusion when troubleshooting traffic anomalies. Adequate documentation is essential before rolling this kind of adjustment into production.

Advisory for Implementation

Prioritize validation in a lower-stakes environment, mimicking realistic workloads. Employing performance testing tools like k6 or Locust can unearth load imbalances resulting from uneven pod allocations. It’s wise to introduce changes incrementally, starting with less critical services, while closely monitoring error rates and p95 latency in addition to cross-AZ metrics.

While locality-aware load balancing isn’t a new feature in Istio, it often goes overlooked in favor of simpler defaults. However, addressing load balancing configurations could reveal significant performance improvements for your multi-AZ deployments.