Spark Executor Sizing on YARN: Why Allocated Memory Can Leave CPU Cores Idle

YARN’s packing problem

Spark clusters can pay for CPU capacity that batch jobs don’t use. A node may appear nearly full on memory while its CPUs sit half idle: YARN reports memory allocated to containers, and the executors placed on a node may not fill all its cores. That mismatch matters because cluster capacity is constrained by more than one resource at a time. A placement that fits the available memory can still leave CPU capacity stranded, while a request that would use the CPUs may not fit in the remaining memory.

This is a resource-packing problem. YARN schedules containers against the resources it tracks, and Spark executors consume those allocations while processing work. The scheduler can’t treat memory and CPU as interchangeable: spare cores don’t make room for a container whose memory request exceeds what’s available, and spare memory doesn’t give an executor more cores than its configuration permits. The result can be a cluster that looks busy by one measure and underused by another.

That distinction can be easy to miss when looking at a single dashboard or a single job. Memory allocation describes what’s been assigned, not necessarily how much of that memory the application is actively using. CPU utilization describes work being done at a point in time, not whether the node could accept another executor under YARN’s placement rules. Both views are useful, but neither explains fit on its own.

What shapes executor fit

Spark requests one YARN container per executor [3]. Each request includes the executor heap, configured by spark.executor.memory, plus off-heap overhead. That overhead defaults to 10% of executor memory, with a 384 MiB minimum [1]. In practical terms, the container request is larger than the heap setting alone suggests. A calculation based only on executor memory can therefore overstate how many executors will fit on a node.

The distinction between heap and container memory is central. The heap is where the executor’s managed Java objects reside; the overhead allowance covers memory use outside that heap, including native and other non-heap demands. Those demands vary with the workload and its runtime behavior, so a configured heap doesn’t describe every memory requirement. If the container is sized too tightly for the executor’s actual use, the nominal fit calculation may look sound while the job still encounters memory pressure.

Heap capacity is split among a reserved area, user data structures, and a unified execution and storage region. The unified region gets 60% of usable heap by default, controlled by spark.memory.fraction [2]. That division shapes how much room remains for operations such as processing data and retaining cached data, but it doesn’t change the size of the YARN container request. Treating the fraction as a way to shrink the container would confuse an internal heap setting with the scheduler’s allocation.

YARN rounds container requests up to its allocation increment, so actual node fit can differ from the nominal calculation [4]. A request just above an increment boundary may receive a larger allocation than its raw memory total implies. That rounding can affect the number of containers that fit, particularly when a node’s remaining resources are close to a boundary. Executor sizing therefore affects how effectively a workload uses both memory and cores.

There’s a second dimension: how many cores an executor is configured to use. Memory determines whether a container can be placed; the executor’s core allocation determines how much CPU capacity that executor can claim. A setup with relatively large containers and few cores per executor can leave some node cores unused after the available memory is assigned. A setup with smaller containers may allow more executors to fit, but that doesn’t automatically mean the workload will run better. The right balance depends on the job’s memory needs and how it uses parallel work.

Why the arithmetic can mislead

A simple packing estimate starts with the node’s allocatable memory, divides it by the rounded container request, and compares the result with available cores and the executor core setting. That gives a useful upper bound, not a promise about actual placement. YARN may have other allocations on the node, and a job can encounter a different amount of free capacity when it asks for containers than a static calculation assumes.

Nor does a higher executor count guarantee higher throughput. More executors can expose more parallelism, but they also create more containers that must fit within the cluster’s resources. Smaller allocations can leave less heap per executor; larger allocations can reduce the number of placements. If the workload’s active tasks don’t need every assigned core, increasing CPU allocation may simply increase idle capacity. The same configuration can therefore look efficient under one workload and wasteful under another.

(And this is the part most people overlook.) A node’s “full” memory reading may mean that memory has been reserved for containers, not that every byte is being used by Spark’s heap. Conversely, a low CPU reading doesn’t prove that the scheduler can place another executor there: the remaining memory might be too small, or the container request might round up beyond what’s available. The resource counters describe different constraints.

What operators can learn from fit

If you’re working in this space, compare the configured container request with the resources YARN can actually allocate on a node, then check whether the executor’s cores make use of the CPU capacity that remains. Include the overhead allowance and allocation increment in that comparison. This is more significant than it looks: an apparently minor difference between heap size and container size can change how many executors fit, and a count of containers alone won’t show whether their CPUs are busy.

It’s also useful to look at the workload’s behavior over time rather than relying on one snapshot. Batch jobs can have phases with different memory and CPU demands. A node that is well packed during one phase may show idle cores during another, while a memory-heavy phase may restrict placement even when CPU use is modest. No single executor size can eliminate every mismatch across changing stages.

The trade-off is straightforward, even if the tuning isn’t. Larger executors may provide more room within each process, but they consume larger container allocations and can be harder to place. Smaller executors may fit more readily, yet bring their own per-container demands and may not suit every workload. There’s no universal size implied by the default settings; they’re inputs to a fit calculation that should be checked against the job and the cluster.

Implications for cluster efficiency

YARN’s packing problem is a reminder that allocated capacity and useful work aren’t the same thing. Memory, cores, container rounding, and executor configuration interact, so a cluster can have idle CPU even while memory limits prevent another placement. The numbers here can be underwhelming: a tidy allocation report doesn’t by itself demonstrate efficient use.

What this means for you is that executor sizing should be judged by both placement and workload behavior. Check how much memory each request consumes after overhead and rounding, how many executors can fit, and whether those executors keep their assigned cores occupied. Then revisit the calculation when the workload or cluster allocation changes. A better fit can reduce obvious waste, but it won’t guarantee that every resource stays busy at every moment. Sometimes, the constraint is the shape of the request.