The Hidden Risks of Kubernetes Readiness Probes in Rolling Updates

Introduction

Kubernetes rolling updates are designed to facilitate smooth application deployments without downtime. However, the reality can sometimes contradict the theory—especially when readiness probes only verify process health instead of true application readiness. Understanding the fundamental disconnect can help streamline deployment processes and enhance application reliability.

The Problem with Readiness Probes

During a recent deployment of a Kubernetes-based 5G signaling service, the situation reflected a common oversight: every readiness probe signaled success while user sessions were actually failing. Kubernetes marked new pods as healthy, yet an 18% increase in session establishment failures indicated something amiss during the update process.

This discrepancy illustrates how readiness probes, typically configured for HTTP health checks, may yield a misleading status of operational readiness. The failure to consider the time required for essential preliminary steps—like completing protocol handshakes and registering with upstream systems—can lead to cascading issues during rolling updates.

Understanding Readiness Probes

At their core, Kubernetes readiness probes serve a critical function. They ascertain whether a pod is ready to receive traffic. The typical configuration involves sending an HTTP GET request to a defined endpoint and awaiting a 200 OK response before allowing traffic flow. This method works effectively for stateless applications relying solely on HTTP.

However, when dealing with stateful services, such as those that communicate via SIP or Diameter, the standard HTTP checks fall short. These services undergo several processes to become truly "ready," such as registering with authentication services or establishing peer relationships before they can process traffic effectively. The inherent delays in these actions create a gap between process initiation and genuine readiness.

What Went Wrong in Production

In the 5G signaling service deployment, the readiness probe was configured around a lightweight HTTP endpoint that confirmed the process startup by returning a 200 response just seconds after booting. This oversimplification obscured the actual state of the pod, which needed additional time to complete SIP registration and re-establish its session state.

As a result, while Kubernetes marked these pods as operationally ready, the underlying telemetry revealed that they could not effectively handle traffic reliant on registered signaling states. The ensuing session failures were both subtle and damaging, yet the visibility of infrastructure metrics painted an overly optimistic picture of deployment success.

Protocol-Aware Readiness Checks

The solution to this readiness verification problem involves the adoption of protocol-aware checks. Unlike standard HTTP probes, these checks can verify that an application has not only started but is also fully integrated within its operational context.

In the case of the 5G signaling service, transitioning from an HTTP probe to an exec probe that checks registration status proved effective. The configuration might appear as follows:

readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - "check-registration-status.sh && exit 0 || exit 1"
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 12

In this setup, the script actively verifies if the pod has successfully registered with upstream systems before indicating readiness. This change provides a more accurate reflection of the pod's state, eliminating false positives from standard checks.

Applying the Lessons Beyond Telecom

The techniques derived from this example are valuable beyond telecommunications. Any application that relies on stateful communication, warm connection pools, or protocol-dependent registrations should adopt similar readiness-check methodologies. This shift ensures that your traffic is routed only to truly prepared instances, avoiding unexpected failures and enhancing overall reliability during deployment.

Concluding Thoughts

The distinction between being "running" and "ready" is crucial in environments utilizing Kubernetes. While HTTP probes serve as a good starting point, they are not suitable replacements for more nuanced readiness evaluations required by complex applications. If teams continue to rely solely on basic HTTP checks, they're merely achieving zero-visible-downtime rather than genuine zero-downtime deployments. Ultimately, ensuring your application meets its specific operational criteria before handling traffic is essential to maintaining a high level of service availability.