Kubernetes-Native ML Platforms Need Declarative Lifecycle Management

Kubernetes can schedule GPU workloads and restart failed containers. Those capabilities solve important infrastructure problems, but they don’t make an ML platform Kubernetes-native by themselves. The deeper test is whether teams can manage the machine learning lifecycle through declarative state: setting an intended outcome, observing progress and reconciling failures or drift.

That distinction is becoming more consequential as AI workloads move onto Kubernetes. As Vara Bonthu described in a CNCF article about AI’s migration to Kubernetes, training jobs, inference services and GPU scheduling are increasingly part of the cluster’s workload mix. Yet many platforms still coordinate training, evaluation, model registration and deployment through separate APIs, databases and scripts. Kubernetes runs the compute, while custom glue holds the lifecycle together.

My read is that the operational opportunity lies in extending Kubernetes’ control model to the work surrounding the containers. A model’s path from data and code to evaluation and production should be visible as connected, durable state. That gives teams a practical way to understand what happened, recover from interruptions and make changes with a clearer view of their consequences.

ML platforms manage state over time

A production ML system isn’t just a sequence of jobs. Each training run depends on inputs such as a dataset, code version and parameters; it produces artifacts and results that may be evaluated, registered and deployed. A deployment, in turn, points to a particular model version and has its own rollout policy, health signals and traffic allocation.

Those relationships need to persist through process restarts, infrastructure problems and changes in the teams or tools operating the platform. Treating each step as an isolated batch job makes it harder to reconstruct lineage or determine which part of the process needs attention. The platform needs to represent the lifecycle as state that can be observed and managed continuously.

Kubernetes controllers offer a familiar pattern for that work. A user or automation declares a desired state; a controller compares it with current conditions, takes action and reports progress. The loop repeats, so recovery and correction become part of normal operations rather than special procedures triggered only after a failure.

A rollout makes the pattern concrete

Suppose an operator wants model version 42 to receive 10% of production traffic. The desired allocation can be recorded as intent. If the version currently receives no traffic, a controller can create or update the runtime resources needed to begin a canary rollout, check health signals and report each transition.

The controller’s status should make the current situation legible. It might show that the rollout is waiting for capacity, has failed a policy check or has encountered an unhealthy model. If the signals support continued rollout, it can keep reconciling toward the requested traffic split. If they don’t, the status gives an operator a useful place to start investigating.

This approach also supports measured changes across clusters. A control plane can roll out an update in stages and stop when a problem appears, limiting how far the change travels. Custom Kubernetes resources and controllers let platform teams apply the same pattern to ML concepts such as training runs, evaluations and deployments, while giving those concepts APIs that automation can inspect.

Make lifecycle objects useful to operators

When a platform gives lifecycle concepts explicit resources, their relationships become easier to follow. A training run can point to its inputs and produce a model object containing lineage and evaluation results. A deployment can reference that model and report its health and traffic allocation. Operators can then investigate a lifecycle object and its status instead of piecing together context from unrelated services.

Several operational benefits follow:

• Recovery becomes repeatable: Controllers keep comparing desired and observed state, so transient errors and retries fit into the ordinary operating model.
• Status has a consistent shape: Conditions can show whether a resource is pending, progressing, ready, degraded or blocked, across different stages of the lifecycle.
• Policy can be applied at the right point: Admission controls can check fast requirements such as resource limits and data restrictions, while controllers handle longer-running evaluations and approval gates.
• Automation gets stable APIs: User interfaces, command-line tools, controllers and AI agents can work against versioned, typed resources rather than inventing separate integrations.
• Observability has more context: Metrics and logs can be associated with the lifecycle object that explains the work, such as an evaluation for a particular model candidate.

Connect execution tools through a control plane

There’s little value in rebuilding every training, data processing and inference tool inside one platform. Those engines often have specialized capabilities and evolve on different schedules. A control plane can instead define lifecycle intent, relationships, policy and status, while execution engines handle the actual workloads.

Adapters bridge the two layers. They translate the control plane’s intent into the resources that a given runtime understands, then return results to the shared lifecycle model. That structure gives users a consistent way to manage work while leaving room for different engines and workload requirements. It also makes it easier to change an execution component without redesigning the whole user experience.

The resource model needs restraint, though. Kubernetes APIs shouldn’t become a dumping ground for internal details or every short-lived function call. A good resource represents durable intent that a user or automation system needs to manage and observe, such as a model deployment. Large model files, logs, traces and detailed evaluation output belong in storage designed for those artifacts; the Kubernetes resource can hold references and concise status.

Before adding a resource type, platform teams can ask whether it captures meaningful intent, persists beyond one process, needs ongoing status tracking and benefits from reconciliation. The API should also be stable enough to serve as a lasting contract. If those conditions don’t apply, putting the object in the cluster may add complexity without improving operations.

Keep the contract durable as AI workloads change

Experience with Uber’s ML platform, Michelangelo AI, helped make the value of this separation clear. Its modular control plane has been released as an open-source project. The enduring abstraction, in this view, isn’t a specific training engine, serving runtime or workflow implementation. Those components vary and change. The lifecycle contract connecting them can remain stable.

That idea applies beyond conventional machine learning. Generative and agentic systems add prompts, tools, policies, benchmark suites and traces to the set of artifacts a platform must track. The operational questions still sound familiar: What did the system receive as input? Which versions did it use? What happened? Did it meet evaluation and policy requirements? Can the result be reproduced, promoted or rolled back?

These questions call for explicit state and lineage. A platform that records them through consistent lifecycle resources can give operators a clearer account of how a result was produced and what is running now, even as the underlying execution tools change.

Shared patterns are the next platform task

Kubernetes provides a common foundation for running AI workloads. The next area for open-source collaboration is the set of patterns used to manage their lifecycles. Workloads differ too much for a single rigid MLOps API, but teams can still benefit from conventions for lineage, status conditions, portable workload definitions, identity and policy boundaries, evaluation and promotion, and observability across compute, accelerators and storage.

The practical measure of a Kubernetes-native ML platform is whether it can declare, observe and reconcile the work from training through production. As AI systems accumulate more components and artifacts, platforms that preserve that lifecycle context will make changes easier to track, failures easier to diagnose and recovery easier to repeat.

KubeCon + CloudNativeCon is scheduled for Salt Lake City from November 9 to 12. Register Now!