Kubernetes v1.36 Empowers Controllers with Server-Side Sharded List and Watch Feature
Kubernetes clusters have reached capacity challenges as they scale to tens of thousands of nodes, particularly affecting controllers that monitor high-cardinality resources like Pods. Traditionally, each controller replica consumes substantial CPU and memory resources by processing the entire event stream from the API server, even for events that don’t pertain to its duties. This setup doesn't alleviate the resource drain; instead, it amplifies it.
The latest Kubernetes v1.36 introduces an alpha feature, server-side sharded list and watch, aimed at addressing these inefficiencies (see KEP-5866). When activated, this feature allows the API server to filter events before they reach the controller replicas, ensuring that each replica only processes the events relevant to its designated resource collection.
Limitations of Client-Side Sharding
Some existing controllers like kube-state-metrics leverage horizontal sharding, dividing responsibilities across replicas. However, this method doesn’t reduce the overall data load from the API server. Each replica still processes every event, resulting in:
- N replicas x full event stream: Each replica deserializes all events, discarding the irrelevant ones.
- Network usage scales with the number of replicas, rather than the size of the shard.
- CPU cycles are wasted on deserialization of discarded events.
With the server-side sharded list and watch, the API server takes on the role of filtering, leading to a more efficient process. Each replica specifies which hash range it is responsible for to the API server, which only relays relevant event notifications.
Mechanism of Operation
This feature introduces a new shardSelector field in ListOptions. Clients designate a hash range using the shardRange() function:
shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000')The API server generates a deterministic 64-bit FNV-1a hash of the field and returns only those objects whose hash fits the specified range. This applies similarly to list responses and watch event streams, allowing for consistency across different API server instances.
Currently, the fields available for hashing include object.metadata.uid and object.metadata.namespace.
Implementing Sharded Watches in Controllers
Controllers typically implement informers for resource list and watch functionality. To incorporate sharding, each replica introduces the shardSelector within the ListOptions using WithTweakListOptions:
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/informers"
)
shardSelector := "shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000')"
factory := informers.NewSharedInformerFactoryWithOptions(client, resyncPeriod, informers.WithTweakListOptions(func(opts *metav1.ListOptions) {
opts.ShardSelector = shardSelector
}))For a deployment with two replicas, sharding divides the hash space:
// Replica 0: lower half of the hash space
"shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000')"
// Replica 1: upper half of the hash space
"shardRange(object.metadata.uid, '0x8000000000000000', '0x10000000000000000')" A single replica can also cover non-contiguous ranges by using ||:
"shardRange(object.metadata.uid, '0x0000000000000000', '0x4000000000000000') || " + "shardRange(object.metadata.uid, '0x8000000000000000', '0xc000000000000000')" Checking Server Capability
To confirm that the API server acknowledges the shard selector, the list response will feature a shardInfo field in the metadata, illustrating the applied selector:
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"resourceVersion": "10245",
"shardInfo": {
"selector": "shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000')"
}
},
"items": [...]
}If the shardInfo field is missing, the shard selector wasn’t honored, and the client received an unfiltered collection. In this situation, clients should be equipped to filter results client-side based on their assigned shard range.
Participate in the Development
The feature is currently in alpha mode and requires the enabling of the ShardedListAndWatch feature gate on the API server. Feedback from controller developers and operators managing large clusters is encouraged.
For discussions or comments, join the #sig-api-machinery channel on Kubernetes Slack.
