EKS Cost Optimization: Why Kubernetes Clusters Leave CPUs and GPUs Idle

EKS gives enterprises a broad menu of compute options, but the scale of that menu hasn’t solved a basic operational problem: much of the capacity Kubernetes teams pay for sits idle. The 2026 State of Kubernetes Optimization Report, based on production measurements across tens of thousands of clusters and millions of compute resources, points to a widening gap between provisioned infrastructure and actual workloads.

Across AWS, Azure and GCP, average CPU utilization fell from 10% to 8% in 2025, while memory utilization dropped from 23% to 20%. CPU overprovisioning rose from 40% to 69%; memory overprovisioning reached 79%. GPU utilization averaged 5%, including on EKS. Those figures describe clusters before automated optimization was activated, giving teams a view of ordinary operations rather than a tuned benchmark.

Karpenter can provision quickly, but requests set the floor

Karpenter has helped teams move past some limits of the native Cluster Autoscaler and fixed EC2 Auto Scaling Groups. It can respond to pending pods and provision EC2 instances without requiring teams to maintain prebuilt node groups. That flexibility matters, but it doesn’t change the signal Kubernetes gives the scheduler: resource requests.

If a service requests 4 vCPUs and 8 GiB of memory, Karpenter provisions to meet that declared need. The application might use only 500 millicores and 1.5 GiB at peak. The unused capacity remains part of the instance footprint all the same.

Teams often set requests defensively to avoid CPU throttling, latency problems or out-of-memory (OOM) evictions. Because generous requests rarely trigger an immediate outage, they can become defaults embedded in Helm charts and Terraform templates, then spread across environments. The cluster may satisfy its Kubernetes configuration while remaining poorly matched to application behavior. On EKS, organizations managing their own EC2 nodes pay for that unused headroom.

GPU sharing changes the economics of inference

Low GPU utilization is especially expensive: an idle GPU can cost dollars per hour. In January 2026, AWS raised H200 Capacity Block prices by 15%, citing supply and demand imbalances. The report still found average EKS GPU utilization at 5%, leaving a large share of paid accelerator capacity unproductive.

One reason is the dedicated allocation model. Inference traffic can arrive in bursts, with long quiet periods between requests. If an endpoint has exclusive use of a GPU instance, its owner pays for the full capacity even when the model is waiting.

Edtech platform ALLEN Digital illustrates another approach. The company ran seven machine learning models, including BGE-M3, LlamaGuard and Multilingual E5 Large. It moved them from dedicated Amazon SageMaker instances to EKS, where GPU time-slicing and automated bin-packing allowed multiple models to share physical GPUs. With a 50/50 mix of Spot and on-demand capacity, ALLEN Digital reported compute costs more than 70% lower than on SageMaker, with no compromise in inference latency.

The practical shift is from assigning a GPU to a model around the clock to sharing accelerator time across workloads whose demand doesn’t peak together. Better packing can reduce the number of cards needed to serve the same work.

Spot savings depend on where workloads run

EC2 Spot Instances can cost up to 90% less than on-demand capacity, yet fewer than 2% of GPUs in the report ran on Spot. Teams often avoid Spot for production because instances can receive a two-minute interruption notice. The report’s regional data suggests that treating interruption risk as uniform can leave savings on the table.

For NVIDIA T4 instances (g4dn.xlarge), Spot survival probability in eu-west-3 remained above 90% over a 24-hour period. In us-east-1 and eu-central-1, it fell below 20% in that same window. Regional pricing also varied: for p4d.24xlarge A100 instances, Spot prices differed by between two and nearly five times across us-east-1, us-east-2 and us-west-2. The report says shifting workloads in response to market conditions produced savings of up to 80% against the average Spot price.

That variability makes manual tracking of prices, availability zones and interruption patterns difficult to sustain. Iterable, a marketing platform, used automated hourly Spot rebalancing across its AWS environment. The company removed 2,000 unneeded CPUs, reduced its node count by 10% and cut compute spend by 75% while managing real-time traffic swings.

Graviton adoption is growing alongside stateless workloads

ARM CPU nodes are growing 3.5 times faster than x86 nodes, measured from the second quarter of 2024, and now represent 9% of the Kubernetes CPU fleet in the report. That expansion aligns with workload composition: 93% of observed deployments were stateless.

Stateless APIs, microservices and background workers can often run on Linux ARM64 without persistent-volume or storage I/O constraints. Moving an oversized workload to Graviton can reduce per-core costs, but the savings stack up when teams first correct the workload’s CPU and memory requests. Rightsizing removes capacity the application doesn’t use; a suitable instance family can then lower the cost of the capacity it does need.

More headroom doesn’t automatically mean more reliability

Large resource buffers are often justified as protection for service-level objectives. But static overprovisioning doesn’t necessarily protect the workloads under pressure. In one production environment examined by the report, OOM kills averaged 40 to 50 per measurement window and sometimes exceeded 80, despite extensive resource padding.

Broad buffers can leave quiet services with excess capacity while fast-growing or memory-leaking applications continue using stale limits. Automated rightsizing can lower allocations for workloads that consistently run below their requests and raise limits when it detects memory pressure. In the environment described, this approach brought OOM kills close to zero while reducing provisioned CPU by about 50%.

The finding challenges the assumption that teams must choose between efficiency and reliability. Resource settings based on observed behavior can remove waste while also responding to pressure that static padding misses.

Rightsize before locking in cloud commitments

Savings Plans and Compute Reserved Instances can reduce rates, but a long commitment made before optimization can discount capacity applications still don’t use. With CPU overprovisioning at 69% in the report, committing to the existing footprint risks baking that mismatch into future spending. The article says manually managed commitments can leave up to 35% of prepaid resources unused as workloads and traffic change.

A practical sequence starts with workload requests grounded in production consumption. Teams can then pack nodes more efficiently, consider instance families such as Graviton, and place interruption-tolerant work on Spot with multi-availability-zone coverage and on-demand fallback. Commitments should reflect the baseline that remains after those changes; the article recommends keeping utilization of that committed baseline near 98%.

The report’s figures span EKS, AKS and GKE, suggesting that low utilization isn’t unique to one cloud provider. The operational challenge grows as fleets spread across teams, regions and instance types. EKS offers plenty of ways to change where work runs and how capacity is allocated. The harder task is keeping those decisions aligned with what applications actually consume, as demand shifts.