Kubernetes v1.36: Enhancements to Mixed Version Proxy for Safer Upgrades

With Kubernetes 1.36 now making its debut, the Mixed Version Proxy (MVP) transitions from alpha to a default beta feature, enhancing the safety of cluster upgrades. Originally introduced in version 1.28, MVP addresses the critical issue of resource misrouting during active upgrades when different API servers handle different versions.

Understanding the Challenge

In environments with high availability, multiple API servers often operate different versions simultaneously, which can lead to complications. For instance, if a request is sent to an older API server that doesn’t support a new resource introduced in an upgrade, it will erroneously respond with a 404 Not Found error. This situation can cause problems such as mistakenly triggering garbage collection or preventing namespace deletions. MVP resolves this by intelligently proxying requests to the correct API server capable of handling them.

sequenceDiagram
participant Client
participant API_Server_A as API Server A (Older/Different)
participant API_Server_B as API Server B (Newer/Capable)
Client->>API_Server_A: 1. Request for Resource (e.g., v2)
Note over API_Server_A: Determines it cannot serve locally
API_Server_A->>API_Server_A: 2. Looks up capable peer in Discovery Cache
API_Server_A->>API_Server_B: 3. Proxies request (adds x-kubernetes-peer-proxied header)
API_Server_B->>API_Server_B: 4. Processes request locally
API_Server_B-->>API_Server_A: 5. Returns Response
API_Server_A-->>Client: 6. Forwards Response

Evolution of Mixed Version Proxy

The MVP’s journey from its initial implementation to the present has involved significant upgrades and architectural improvements. Here’s a closer look:

1. Shift from StorageVersion API to Aggregated Discovery: Initially, MVP relied on the StorageVersion API for determining peer resources, which had limitations, particularly with Custom Resource Definitions (CRDs). In the latest version, this has changed to utilize Aggregated Discovery, allowing servers to dynamically assess capabilities among peers.

2. Introducing Peer-Aggregated Discovery: Early MVP implementations lacked a mechanism to combine local and peer discovery requests. Version 1.36 enhances this with Peer-Aggregated Discovery, enabling a unified view of available API resources across the cluster when requests for resource discovery are made.

sequenceDiagram
participant Client
participant API_Server_A as API Server A
participant API_Server_B as API Server B
Client->>API_Server_A: 1. Request Discovery Document
API_Server_A->>API_Server_A: 2. Gets Local APIs
API_Server_A->>API_Server_B: 3. Gets Peer APIs (Cached or Direct)
API_Server_A->>API_Server_A: 4. Merges and sorts lists deterministically
API_Server_A-->>Client: 5. Returns Unified Discovery Document

It’s important to note that while peer-aggregated discovery will be the default behavior, nodes can request a non-aggregated view of their local API resources using the profile=nopeer parameter in their request headers.

Essential Configuration Requirements

The transition of MVP to beta in 1.36 means it’s essentially a requirement for Kubernetes users to ensure their API servers are appropriately configured:

• --feature-gates=UnknownVersionInteroperabilityProxy=true: Ensure that this feature gate remains active. It's confirmed by default in this version.
• --peer-ca-file=<path-to-ca>: This is essential. You must specify the CA bundle for authenticating the certificates of peer API servers; without it, requests will falter due to TLS issues.
• --peer-advertise-ip and --peer-advertise-port: These enable configuration for how peers communicate with the API server, especially vital in complex network architectures.

Using kubeadm for Configuration

If your cluster operates under the management of kubeadm, these flags should be incorporated within your ClusterConfiguration file:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
apiServer:
 extraArgs:
 peer-ca-file: "/etc/kubernetes/pki/ca.crt"
 # specify peer-advertise-ip and port as necessary

Next Steps

For clusters with multiple masters undergoing regular upgrades, the rollout of MVP presents a substantial safety enhancement. Moving to 1.36 enables administrators to:

1. Double-check API server flags to confirm the --peer-ca-file is correctly configured.
2. Conduct tests within staging environments as you gear up for the upgrade.
3. Share feedback with SIG API Machinery to enhance future iterations, either via Slack, mailing list, or through participation in upcoming SIG meetings.