The increasing integration of AI, edge computing, and advanced telecommunications within Kubernetes is prompting a reconsideration of how we manage hardware resources. Traditional metrics focusing solely on CPU cycles and memory allocations are proving insufficient. Today, we must account for a range of specialized hardware, including GPUs, TPUs, and network interfaces. This need often arises after a pod has started and can involve complex scenarios like time-sharing among various workloads.
To tackle these challenges, the **Device Management Working Group** within the Kubernetes ecosystem was formed. Their flagship initiative, **Dynamic Resource Allocation (DRA)**, has just achieved General Availability (GA), symbolizing a transformative advance in how Kubernetes manages hardware-dependent applications at scale.
In an upcoming dialogue, we’ll engage with the leaders of the working group — **Kevin Klues**, **Patrick Ohly**, and **John Belamaric** — to delve into the shortcomings of the previous device management model, the intricate NP-hard scheduling challenges, and their vision for a future where Kubernetes can effectively accommodate a more diverse array of hardware resources.
Introducing Device Management
**Natalie Fisher**: Can you provide a brief introduction of yourselves, your roles, and how you became involved with the Device Management Working Group?
**Kevin Klues**: I’m Kevin Klues, a Distinguished Engineer at NVIDIA, and I’ve been co-chairing the Device Management Working Group since its inception at KubeCon EU 2024. I also played a key role in the development of DRA, which began around 2019/2020. Alongside my work on DRA, I’ve been a kubelet maintainer focusing on various device management components since 2019. Our decision to initiate DRA stemmed from encountering significant hurdles while trying to provision external accelerators, such as GPUs, using the existing Kubernetes framework.
**Patrick Ohly**: I’m Patrick Ohly, a Principal Engineer at Intel. Within Kubernetes, I serve as the Tech Lead for **SIG Testing** and **SIG Instrumentation** and co-chair the Device Management Working Group as well. My early contributions include working on ephemeral CSI volumes and storage capacity tracking, giving me insight into API design and implementation. The journey toward developing a new API for accelerators was met with skepticism initially. It was only in 2023 that the community began recognizing the need for this work, which ultimately led to the establishment of our working group.
**John Belamaric**: I’m John Belamaric, a Senior Staff Software Engineer at Google and one of the early co-chairs of the Device Management Working Group. My involvement aligned with my role as co-chair of **SIG Architecture** since 2019. As Patrick mentioned, by late 2023, there was renewed interest in DRA due to its complex implementation. I joined to help resolve community concerns, leading to a collaborative effort that solidified the group's formation following discussions at KubeCon in Paris in 2024.
The Problem and the Solution
The formation of this working group is rooted in a critical reevaluation of how Kubernetes interacts with specialized hardware. At the center of this transformation is **Dynamic Resource Allocation (DRA)**. Instead of merely treating devices as abstract integers, DRA introduces a sophisticated framework for managing device interactions through four key stages:
1. **Modeling**: Vendors utilize the **ResourceSlice API** to clearly communicate their hardware’s capabilities.
2. **Requesting**: Users articulate their hardware needs, such as GPU memory, via the **ResourceClaim API**.
3. **Scheduling**: Kubernetes intelligently matches workload demands with available hardware.
4. **Actuation**: The system executes the necessary steps to prepare the hardware for a pod's operation.
**NF**: For those unfamiliar with it, what is the Device Management Working Group, and what challenges does it seek to address?
**KK**: The Device Management Working Group exists to streamline the configuration, sharing, and allocation of accelerators and specialized hardware within Kubernetes workloads. This includes devices like GPUs, TPUs, and FPGAs, which do not align with Kubernetes’ conventional resource allocation methods.
The challenge lies in the limitations of the old Device Plugin API, which reduces device requests to mere integers. For instance, if you ask for "2 GPUs," you're not conveying how those GPUs should be used, how they interconnect, or whether they can be shared or partitioned. While this was sufficient for basic scenarios, the landscape of modern AI/ML workloads demands much more complexity, as they often require specific interconnects and dynamic sharing of hardware across multiple nodes.
Our primary deliverable, Dynamic Resource Allocation (DRA), redefines this rigid approach through a flexible, declarative API. With DRA, workloads can specify their exact hardware needs, and vendors can provide detailed device attributes that the Kubernetes scheduler can leverage. DRA officially moved to GA with Kubernetes 1.34, sparking significant growth in related tools and drivers.
**PO**: As Kevin noted, our group began with a limited number of contributors to bring DRA to fruition. Given its wide-ranging implications for various areas of Kubernetes, we recognized the need for a community hub where device vendors, Kubernetes maintainers, and end-users could collaborate and contribute. Regular meetings and discussions, both online and during KubeCon, help maintain that flow of dialogue.
**JB**: So far, our work has centered around DRA, focusing on the selection, allocation, and configuration of hardware devices. We broke down the problem into four parts: how vendors model their devices, how users articulate their requests, how we efficiently schedule those requests, and how we prepare devices for use by pods.
An essential aspect of our approach is acknowledging the diversity of hardware and the rapid evolution of the hardware landscape. Instead of adjusting Kubernetes APIs for each new hardware type, we’ve aimed to create a flexible infrastructure that accommodates various devices through a combination of scheduling and configuration. However, DRA still leans heavily on scheduling. We must address other device management aspects, such as failure detection and support for complex hardware interconnections.
These diversification efforts pave the way for a more comprehensive solution, addressing how various devices align and function collectively, which remains a key area for future development.
A Cross-SIG Effort
Given the extensive scope of device management, which encompasses scheduling, node operations, autoscaling, networking, and API design, it became essential for the Device Management Working Group to collaborate with multiple SIGs across Kubernetes.
**NF**: How does this collaboration across SIGs actually function, and what makes it crucial?
**KK**: The Device Management Working Group acts as a connective layer, integrating multiple layers of Kubernetes. We represent five key SIGs: sig-node, sig-scheduling, sig-autoscaling, sig-network, and sig-architecture.
Our group doesn't directly control code; instead, we develop KEPs and implementations within the respective SIGs. What we provide is a forum that encourages cross-disciplinary collaboration, enabling those working on scheduling, kubelet functionality, autoscaling, and networking to share insights and designs rather than working in isolation.
Consider a simple yet telling example: if a user requests GPUs that need to function through NVLink, that requirement impacts scheduling, kubelet configuration, and possibly autoscaling. If each SIG operates independently, we risk creating inconsistent frameworks, duplicative logic, and, ultimately, integration issues that only manifest when put into production. Our mission is to facilitate a coherent API and data model across these components.
Moreover, this cross-SIG approach ensures diverse perspectives on design decisions. A contributor from sig-scheduling may highlight complexities that a sig-node member might overlook, and vice versa. While this slows down the decision-making process somewhat, it leads to more resilient and effective outcomes.
Current Focus Areas
With DRA achieving GA status, the focus of the working group has shifted toward enabling more sophisticated scheduling models, improving operational visibility, and tackling increasingly complex hardware configurations.
**NF**: What initiatives or deliverables is the working group currently prioritizing?
**KK**: We keep an updated project board at the [Kubernetes Project Board](https://github.com/orgs/kubernetes/projects/95) to track our ongoing initiatives and progress in real-time.
**PO**: Our initial DRA framework was intentionally narrow in scope to allow us to reach GA effectively. However, we are now developing additional KEPs that will increase our feature set. These can be broadly categorized into three goals:
1. Expand DRA's capabilities to accommodate more intricate devices and scheduling scenarios.
2. Enhance day-two operations, such as health monitoring.
3. Improve support for multi-node configurations, especially through workload-aware scheduling.
Besides the project board, we maintain a comprehensive table summarizing all active [KEPs](https://www.kubernetes.dev/resources/keps/) currently in the pipeline. Some are slated for the upcoming 1.36 release, with more expected to follow for 1.37.
**NF**: Efficient device utilization and sharing remains a core challenge. What advancements have been made in this area?
Summary and Future Implications
The ongoing innovations in Kubernetes’ device management reflect a decisive move towards making high-performance computing and AI/ML workloads more manageable and efficient. The recent developments — particularly through the integration of the Dynamic Resource Allocation (DRA) technology — demonstrate not just a technical evolution but also a paradigm shift in workload orchestration.
What's striking here is how these API enhancements empower users to make more nuanced requests for resources, rather than being confined to a rigid and potentially inefficient selection of hardware. The flexibility to specify resource requirements, like "at least 20GB of memory," rather than demanding a particular model, is a big deal. It not only broadens the availability of hardware but optimizes its utilization, which is critical in resource-intensive environments.
And yet, it's not all smooth sailing. The complexity introduced by these expanded capabilities begs the question: how do we ensure that the enhancements lead to better outcomes rather than just complicating the scheduling processes further? The balancing act between flexibility and optimal resource allocation is a significant hurdle that the Kubernetes community is keenly aware of. Efficient scheduling remains a complex challenge, one that requires ongoing attention as the number of contributing developers grows.
Looking ahead, the excitement is palpable within the community, particularly following NVIDIA's donation of essential drivers to the open-source project. Such collaborations signal a promising future where experts can coalesce around shared tools and frameworks. The potential for innovative applications, especially as these technologies invite experimentation beyond traditional use cases, is quite vast.
If you’re involved in this space, now is an opportune moment to jump in. The landscape is fertile for contributions, not just in development and technical design, but also in shaping how we think about hardware orchestration. Engaging with the Device Management Working Group is a chance to influence the trajectory of Kubernetes as it evolves to meet the demands of tomorrow’s computing challenges. Whether your strengths lie in coding, API design, or community building, there’s a place for you to leave your mark in this endeavor.
