Kubernetes v1.37: Scheduler Preemption for In-Place Pod Resize (Alpha) (kubernetes.io)

by rss-bot · 2 months ago · 0 comments

In Kubernetes, resource allocation has historically been a static decision made during a Pod's initial scheduling and placement. With the graduation of the core <a href="https://kubernetes.io/blog/2025/12/19/kubernetes-v1-35-in-place-Pod-resize-ga/">in-Place Pod resize</a> feature to General Availability in v1.35, application developers and cluster operators gained the powerful ability to dynamically adjust CPU and memory allocations of running containers without incurring disruptive restarts or application downtime.
However, in-place resizing introduced a unique resource scheduling gap: if a running Pod requested a resource scale-up that exceeded the host node's allocatable headroom, the Kubelet was forced to mark the request as <code>Deferred</code>. The Pod would remain parked in this state indefinitely, waiting for resources on the node to naturally free up.
To bridge this scheduling gap, Kubernetes v1.37 introduces scheduler preemption for in-place Pod resize (Alpha), behind the <code>InPlacePodVerticalScalingSchedulerPreemption</code> feature gate.
This feature allows the Kubernetes scheduler to actively free up capacity on a fully-utilized node by preempting lower-priority workloads, enabling the pending in-place resizes of critical, higher-priority applications to succeed.
<h2 id="the-deferred-resize-challenge">The "deferred" resize challenge<a class="td-heading-self-link" href="#the-deferred-resize-challenge" aria-label="Heading self-link"></a></h2>To understand why this preemption mechanism is needed, it is helpful to look at how Kubernetes handles running Pod resizing. When a user or controller (such as the <a href="https://kubernetes.io/docs/concepts/workloads/autoscaling/vertical-pod-autoscale/">Vertical Pod Autoscaler</a>) updates the resource requests of an active container, the Kubelet evaluates whether the underlying node has enough spare allocatable capacity to fulfill the increase.
If the node's resources are fully utilized and cannot satisfy the new limits, the Kubelet sets the container's <code>resizeStatus</code> (reported in the Pod's <code>status.containerStatuses[]</code>) to <code>Deferred</code>. Unlike an <code>Infeasible</code> resize request (which is immediately rejected because it exceeds physical machine boundaries, namespace limit ranges, or admission quotas) a <code>Deferred</code> status indicates that the request is valid but is temporarily unable to be actuated, waiting until node capacity becomes available.
Before the introduction of this preemption mechanism, a Pod's in-place resize scale-up request could become permanently blocked if the node was heavily utilized. Even when a critical application (such as an in-memory database or a real-time web server) required more memory to prevent an imminent out-of-memory (OOM) crash, and the node lacked free capacity, the resize remained <code>Deferred</code>.
In this scenario, cluster administrators had limited choices:
<ol>
<li>Manually evict lower-priority Pods from the node to clear resource headroom.</li>
<li>Rely on the cluster autoscaler to eventually spin up a larger node and reschedule the Pod. However, this is an operation that is highly disruptive and violates the core "no restart" value proposition of in-place scaling.</li>
<li>Rely on a custom autoscaling solution, for example a cluster autoscaler that can trigger dynamic node resizing operations itself.</li>
</ol>
Because the <code>kube-scheduler</code> was unaware of deferred resizes on running Pods, it could not leverage standard priority-based preemption to evict lower-priority workloads and make room for the higher-priority running Pod's resource growth.
<h2 id="why-this-matters">Why this matters<a class="td-heading-self-link" href="#why-this-matters" aria-label="Heading self-link"></a></h2>In production Kubernetes environments, cluster administrators strive to maximize resource utilization and efficiency. A common strategy is to bin-pack unused capacity on not-yet-full nodes with lower-priority workloads, such as batch jobs, background data processing, or best-effort tasks.
Without scheduler preemption for in-place resizing, this created a major operational dilemma. If lower-priority workloads consumed the remaining headroom on a node, higher-priority applications running on that same node would become blocked (<code>Deferred</code>) when they needed to scale up to handle sudden traffic surges or memory spikes. Operators were forced to choose between running low-utilization clusters with idle buffer capacity or risking that critical workloads could not resize when needed.
With scheduler preemption for in-place Pod resize, you can confidently bin-pack unused space across your clusters with lower-priority workloads without worrying about them degrading higher-priority Pods or blocking their scale-up requests. If a high-priority workload requires an in-place resize that exceeds available node capacity, the scheduler automatically preempts the lower-priority Pods to clear headroom. You achieve high cluster utilization and cost efficiency while preserving the responsiveness and reliability of critical services.
<h2 id="architectural-mechanics-how-it-works">Architectural mechanics: How it works<a class="td-heading-self-link" href="#architectural-mechanics-how-it-works" aria-label="Heading self-link"></a></h2>Scheduler preemption for in-place Pod resize integrates directly into the core scheduling cycle to coordinate resources dynamically and safely.
<h3 id="centralized-scheduler-tracking">Centralized scheduler tracking<a class="td-heading-self-link" href="#centralized-scheduler-tracking" aria-label="Heading self-link"></a></h3>The <code>kube-scheduler</code> monitors the cluster for running Pods with a <code>Deferred</code> resize status condition. Normally, Pods with <code>spec.nodeName</code> populated are considered successfully placed and bypass the active scheduling queue. Under this feature gate, the scheduler intercepts Pods carrying the <code>Deferred</code> condition, permitting them to remain in active scheduling evaluations specifically to trigger preemption. The scheduler maintains continuous tracking of these Pods until the Kubelet successfully completes the resize actuation.
<h3 id="single-node-preemption-boundary">Single-node preemption boundary<a class="td-heading-self-link" href="#single-node-preemption-boundary" aria-label="Heading self-link"></a></h3>Unlike placement preemption, which evaluates all nodes in a cluster to find the best scheduling fit, preemption for in-place resizing is strictly localized to the Pod's currently assigned node. The scheduler identifies eligible lower-priority "victim" Pods on the same host and initiates their graceful eviction, freeing up local capacity. Preemption is strictly scoped to the same node where the deferred Pod is running; if a node cannot accommodate the resize even after evicting all eligible lower-priority workloads, the resize remains in the <code>Deferred</code> state.
<h3 id="resource-reservation-safety">Resource reservation safety<a class="td-heading-self-link" href="#resource-reservation-safety" aria-label="Heading self-link"></a></h3>To prevent scheduling races and double-allocation, the scheduler treats resources requested for a resize as already consumed. This enables the Kubelet to actuate the resize once the preemption takes effect.
<h3 id="separation-of-concerns-critical-admission">Separation of concerns & critical admission<a class="td-heading-self-link" href="#separation-of-concerns-critical-admission" aria-label="Heading self-link"></a></h3>When a node is under resource pressure, the Kubelet includes a local mechanism known as the critical Pod admission handler. During initial Pod admission, if a critical system Pod arrives on a node that lacks spare capacity, this local handler can directly evict lower-priority Pods on that node to guarantee admission for the critical workload.
A significant architectural benefit of this new feature is the strict separation of concerns between the Kubelet and the scheduler. Under the <code>InPlacePodVerticalScalingSchedulerPreemption</code> feature gate, the Kubelet's critical Pod admission handler does not perform local preemption checks or trigger local evictions for in-place resizing operations. Instead, the Kubelet defers the request and delegates the preemption decision entirely to the scheduler. This guarantees that a single, centralized orchestrator manages all resize-related preemption logic, respecting global priorities, Pod disruption budgets (PDBs), and graceful termination policies.
<h3 id="managing-competing-updates-races">Managing competing updates & races<a class="td-heading-self-link" href="#managing-competing-updates-races" aria-label="Heading self-link"></a></h3>If a competing, higher-priority resize request is submitted for another running Pod on the same node during an active preemption cycle, the Kubelet prioritizes the higher-priority request. The scheduler is designed to observe these updates and will dynamically trigger a new round of preemption if more capacity is required to fulfill the new state.
<h2 id="node-level-preemption-configuration">Node-level preemption configuration<a class="td-heading-self-link" href="#node-level-preemption-configuration" aria-label="Heading self-link"></a></h2>Administrators and automated controllers (such as a cluster autoscaler) can disable preemption specifically for in-place resizes on particular nodes. This is configured using the new <code>spec.podPreemptionPolicy</code> field in the Node Spec:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">apiVersion: v1
kind: Node
metadata:
 name: batch-workload-node
spec:
 podPreemptionPolicy:
 disableResizePreemption:
 - "cluster-autoscaler.kubernetes.io/disable-preemption"
 - "operator.example.com/policy-override"
</code></pre></div>An example use case for this policy is when a controller would prefer to size down other pods or dynamically adjust the node capacity itself when possible, only enabling scheduler preemption as a last resort.
<h2 id="try-it-out">Try it out!<a class="td-heading-self-link" href="#try-it-out" aria-label="Heading self-link"></a></h2>To utilize scheduler preemption for in-place Pod resize:
<ul>
<li>Your cluster must be running Kubernetes v1.37 or later across both the control plane and all worker nodes.</li>
<li>The <code>InPlacePodVerticalScalingSchedulerPreemption</code> feature gate must be enabled across all control plane components (<code>kube-apiserver</code>, <code>kube-scheduler</code>) and the <code>kubelet</code>.</li>
</ul>
<h3 id="mini-tutorial-observe-resize-preemption-in-action">Mini-tutorial: Observe resize preemption in action<a class="td-heading-self-link" href="#mini-tutorial-observe-resize-preemption-in-action" aria-label="Heading self-link"></a></h3>To see this feature in action locally, you can test scheduler preemption on a single-node <code>kind</code> cluster with constrained CPU headroom.
<h4 id="1-create-a-kind-cluster-with-scheduler-resize-preemption-enabled">1. Create a kind cluster with scheduler resize preemption enabled<a class="td-heading-self-link" href="#1-create-a-kind-cluster-with-scheduler-resize-preemption-enabled" aria-label="Heading self-link"></a></h4>Create a <code>kind</code> cluster configuration file named <code>kind-config.yaml</code> with the <code>InPlacePodVerticalScalingSchedulerPreemption</code> feature gate enabled:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 InPlacePodVerticalScalingSchedulerPreemption: true
</code></pre></div>Create the cluster using this configuration, passing the <code>--image</code> flag to ensure the cluster is running Kubernetes v1.37 (or later):
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">kind create cluster --config kind-config.yaml --image kindest/node:v1.37.0
</code></pre></div>
<div class="alert alert-info" role="note"><h4 class="alert-heading">Note:</h4>Make sure that the node image you specify corresponds to a Kubernetes v1.37 cluster or later (such as <code>kindest/node:v1.37.0</code>). Older Kubernetes releases do not support the <code>InPlacePodVerticalScalingSchedulerPreemption</code> feature gate.</div>
Once your cluster is ready, inspect the node to check how many allocatable CPU cores it has:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu
</code></pre></div>In a standard local <code>kind</code> environment, the output shows 8 allocatable CPU cores:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">NAME ALLOCATABLE_CPU
kind-control-plane 8
</code></pre></div><h4 id="2-create-priorityclasses-and-deploy-pods">2. Create PriorityClasses and deploy Pods<a class="td-heading-self-link" href="#2-create-priorityclasses-and-deploy-pods" aria-label="Heading self-link"></a></h4>Create two PriorityClasses and deploy a low-priority Pod (requesting <code>3</code> CPU) alongside a high-priority Pod (requesting <code>4</code> CPU). Together, these workloads consume <code>7</code> of the <code>8</code> available CPU cores, leaving <code>1</code> CPU of free allocatable headroom on the node.
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># preemption-demo.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: high-priority
value: 1000000
globalDefault: false
description: "High priority workload"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: low-priority
value: 1000
globalDefault: false
description: "Low priority workload"
---
apiVersion: v1
kind: Pod
metadata:
 name: low-priority-pod
spec:
 priorityClassName: low-priority
 containers:
 - name: worker
 image: nginx
 resources:
 requests:
 cpu: "3"
 memory: "500Mi"
 limits:
 cpu: "3"
 memory: "500Mi"
---
apiVersion: v1
kind: Pod
metadata:
 name: high-priority-pod
spec:
 priorityClassName: high-priority
 containers:
 - name: app
 image: nginx
 resources:
 requests:
 cpu: "4"
 memory: "1Gi"
 limits:
 cpu: "4"
 memory: "1Gi"
</code></pre></div>Save this manifest to <code>preemption-demo.yaml</code> and apply it:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">kubectl apply -f preemption-demo.yaml
</code></pre></div>Wait until both Pods are running on the node:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">kubectl get pods
</code></pre></div>Output:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">NAME READY STATUS RESTARTS AGE
high-priority-pod 1/1 Running 0 9s
low-priority-pod 1/1 Running 0 9s
</code></pre></div><h4 id="3-request-an-in-place-scale-up">3. Request an in-place scale-up<a class="td-heading-self-link" href="#3-request-an-in-place-scale-up" aria-label="Heading self-link"></a></h4>Patch the high-priority Pod to increase its CPU request from <code>4</code> to <code>6</code> (+2 CPU delta). Because only <code>1</code> CPU of headroom is free on the node, this resize request exceeds remaining allocatable capacity:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">kubectl patch pod high-priority-pod --subresource resize --patch \
 '{"spec":{"containers":[{"name":"app", "resources":{"requests":{"cpu":"6"}, "limits":{"cpu":"6"}}}]}}'
</code></pre></div><h4 id="4-inspect-the-preemption-event-on-the-low-priority-pod">4. Inspect the preemption event on the low-priority Pod<a class="td-heading-self-link" href="#4-inspect-the-preemption-event-on-the-low-priority-pod" aria-label="Heading self-link"></a></h4>With <code>InPlacePodVerticalScalingSchedulerPreemption</code> enabled, the scheduler intercepts the <code>Deferred</code> resize condition on <code>high-priority-pod</code> and targets <code>low-priority-pod</code> for preemption.
To verify that the scheduler actively preempted the low-priority Pod, inspect its events:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">kubectl get events --field-selector involvedObject.name=low-priority-pod
</code></pre></div>In the event stream (or via <code>kubectl describe pod low-priority-pod</code>), you will see a <code>Preempted</code> event emitted by the scheduler:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">LAST SEEN TYPE REASON OBJECT MESSAGE
5s Normal Preempted pod/low-priority-pod Preempted by pod 97dba925-6b5f-4e2f-99f9-d51c30016586 on node kind-control-plane
5s Normal Killing pod/low-priority-pod Stopping container worker
</code></pre></div><h4 id="5-trace-the-resize-event-lifecycle-on-the-high-priority-pod">5. Trace the resize event lifecycle on the high-priority Pod<a class="td-heading-self-link" href="#5-trace-the-resize-event-lifecycle-on-the-high-priority-pod" aria-label="Heading self-link"></a></h4>Next, inspect the event history on <code>high-priority-pod</code> to observe how the resize progressed from being deferred to successfully completed:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">kubectl get events --field-selector involvedObject.name=high-priority-pod
</code></pre></div>You will observe a sequence of events as the Kubelet coordinates with the scheduler:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">LAST SEEN TYPE REASON OBJECT MESSAGE
33s Warning ResizeDeferred pod/high-priority-pod Pod resize OutOfcpu: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2,"error":"Node didn't have enough resource: cpu, requested: 6000, used: 3950, capacity: 8000"}
32s Normal ResizeStarted pod/high-priority-pod Pod resize started: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}
32s Normal ResizeCompleted pod/high-priority-pod Pod resize completed: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}
</code></pre></div><ol>
<li><code>ResizeDeferred</code>: The Kubelet initially marks the resize request as deferred (<code>Warning</code>) due to insufficient CPU headroom on the node (<code>OutOfcpu</code>).</li>
<li><code>ResizeStarted</code>: Once the scheduler preempts <code>low-priority-pod</code> and capacity is released, the Kubelet accepts the new allocation and begins actuating the resize.</li>
<li><code>ResizeCompleted</code>: The Kubelet successfully updates container cgroup limits via the container runtime without restarting the Pod.</li>
</ol>
Finally, verify that the allocated CPU on the container reflects the new request (appending <code>{"\n"}</code> to the JSONPath query ensures a trailing newline in your terminal):
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">kubectl get pod high-priority-pod -o jsonpath='{.status.containerStatuses[0].allocatedResources.cpu}{"\n"}'
</code></pre></div>Output:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell">6
</code></pre></div>This confirms that the in-place resize succeeded.
<h2 id="getting-involved">Getting involved<a class="td-heading-self-link" href="#getting-involved" aria-label="Heading self-link"></a></h2>This feature represents a major step forward for resource scheduling, bringing enterprise-grade density control and workload prioritization to dynamic resource scaling. We invite cluster operators, platform architects, and developers to enable the <code>InPlacePodVerticalScalingSchedulerPreemption</code> feature gate in their testing environments and share feedback.
If you want to share your experience with this feature, please get in touch with the community via <a href="https://www.kubernetes.dev/community/community-groups/sigs/scheduling">SIG Scheduling</a> or <a href="https://www.kubernetes.dev/community/community-groups/sigs/node">SIG Node</a> channels!

AI News

Kubernetes v1.37: Scheduler Preemption for In-Place Pod Resize (Alpha) (kubernetes.io)

Comments