Buoyant Kinference Brings GPU-Aware AI Inference Management to Kubernetes

GPU scarcity and low utilization can coexist in the same AI infrastructure. Buoyant’s open-source Kinference project aims to address that mismatch with a control plane for coordinating vLLM inference workloads across Kubernetes clusters.

The tool gives platform teams controls for managing performance, cost and compliance across distributed deployments. It can use GPU cache data and queue status to direct work across workloads, while tracking GPU consumption by tenant, group or team.

Managing vLLM across Kubernetes

Kinference works with vLLM, the inference engine developed by researchers at the University of California, Berkeley’s Sky Computing Lab. vLLM combines an inference server for handling network traffic with an engine that uses PagedAttention to improve memory throughput.

Buoyant CEO William Morgan said the control plane is intended to make distributed AI workloads easier to operate and scale on Kubernetes. Teams can assign infrastructure resources to workloads and set service quality expectations, instead of relying only on cloud services that charge by token.

That model gives organizations more direct control over how inference capacity is allocated. It also makes the tradeoffs between workload performance and infrastructure cost easier to manage as teams add AI applications alongside existing services.

Using local capacity, with cloud as an option

Self-hosting appeals to organizations handling sensitive data or working under compliance requirements. Keeping models and inference workloads on infrastructure they control can limit exposure to third-party services. Local execution can also reduce latency for iterative AI applications, Morgan said.

Kinference can route selected inference requests to cloud providers when teams specify conditions for doing so. Those conditions may include a saturated GPU pool, usage by a particular tenant or attributes of an incoming request. The approach lets organizations reserve local capacity for workloads that need it while sending other requests elsewhere when demand exceeds available GPUs.

The control plane also addresses a persistent infrastructure mismatch: GPUs are scarce, yet their utilization in AI environments is often in the single digits, Morgan said. Kinference uses cache information, queueing status and workload controls to help distribute inference across the available capacity.

Tracking consumption by tenant, group or team adds a cost management layer. Operators can see which users and workloads draw on GPU resources, giving them a clearer basis for adjusting allocations and finding underused capacity.

A control problem that will grow with AI workloads

The number of AI applications running on Kubernetes remains unclear, but the operational burden is already taking shape. Platform teams must fit inference into environments that also run established application workloads, using infrastructure that cannot expand without limit.

Kinference’s practical value will depend on how well its workload controls match the performance and policy needs of each organization. Its central premise is straightforward: coordinate inference requests with a view of GPU availability, workload priorities and cost, then use cloud capacity when local resources hit defined limits.

As AI workloads claim a larger share of Kubernetes infrastructure, teams will need to manage them as a portfolio of competing demands. GPU visibility and request routing give operators concrete tools to make those tradeoffs.