All articles
Dev ToolsBy Petru Popa · 6 min read

Gang Scheduling Is Beta, and Beta Means Off

Kubernetes v1.37 graduated the AI training feature nobody gets by default.

Kubernetes v1.37 shipped on 26 August 2026 under the release name Garhwal, with 67 enhancements: 16 graduating to Stable, 23 to Beta, 27 entering Alpha, and one deprecation or removal. For anyone running distributed training on Kubernetes, the item that matters is gang scheduling graduating to Beta — all-or-nothing placement for a set of Pods, expressed through the Workload API and the PodGroup object, delivered as KEP #4671 by SIG Scheduling.

The announcement makes the case well. The default scheduler places Pods one at a time, so a multi-node job can end up partially scheduled: some replicas running and holding accelerators, the rest pending behind capacity that never arrives. The Beta also brings workload-aware preemption, so the scheduler evaluates a PodGroup rather than an individual Pod when deciding what to evict, and PodGroup queueing to keep competing jobs from livelocking against each other.

What the announcement does not say is whether upgrading gets you any of it.

Gang scheduling is Beta with the gate defaulted to false

The feature gate table in the released source tree answers that directly. At tag v1.37.0, GenericWorkload is listed at v1.35 as Alpha with a default of false, and at v1.37 as Beta with a default of false. Beta here is a maturity label, not a switch position.

The gate that pairs with it is worse off. DRAWorkloadResourceClaims — the feature that lets a ResourceClaim belong to a PodGroup instead of being created per Pod — is listed at v1.36 as Alpha and at v1.37 as Beta, both defaulted to false, and its dependency list names DynamicResourceAllocation and GenericWorkload. That is the piece an eight-GPU job actually needs, since without it the ResourceClaim controller will not create a claim from a template for a Pod that belongs to a PodGroup. Two gates, both off, one gated on the other.

CompositePodGroup and PodGroupPreemptionPolicy are new in v1.37 as Alpha, also defaulted to false. Of the four gates covering workload-aware scheduling, a stock v1.37 control plane enables zero.

The naming changed too. The GangScheduling and WorkloadAwarePreemption gates were removed, and GenericWorkload is the single gate that now enables core workload-aware scheduling. Neither removed name appears anywhere in the v1.37.0 feature list, so a 1.36 cluster running the Alpha is passing two flags that the new binaries no longer define.

The mandatory work lands on clusters that never turn it on

This is the part worth budgeting for. The urgent upgrade notes record that the scheduling.k8s.io API group was promoted from v1alpha2 to v1alpha3 and v1alpha2 dropped entirely, alongside a conversion of the DisruptionMode enum field into a struct. The instruction attached is to remove all v1alpha2 objects from the kube-apiserver before performing the cluster update. The promotion of Workload and PodGroup to scheduling.k8s.io/v1beta1 carries the same instruction independently.

Two renames ride along for anyone with manifests: the PodGroup condition PodGroupScheduled became PodGroupInitiallyScheduled, and the PodGroup API replaced PodGroupTemplateRef with WorkloadRef.

So the central claim: for a cluster that experimented with gang scheduling on 1.36, v1.37 is a migration with no default-on payoff — the API version you wrote against is gone, the gate you set is gone, and the feature is still off. That is falsifiable in one file read, below.

The tradeoff the Beta graduation bought

One changelog entry deserves reading twice. Workload-aware preemption was changed to perform a single scheduling attempt with all potential victims removed, which the entry states significantly improves performance but can result in a less optimal choice of preemption victims.

A vendor summary would file that under performance. It is a documented quality regression accepted in exchange for throughput, and on a shared training cluster the victims are other teams' jobs. Preempting the wrong four Pods is not a scheduler latency problem, it is a conversation. Worth knowing before the gate goes on, and it is the sort of detail that lives in a changelog entry rather than in a launch post — the gap why enterprise AI doesn't ship keeps pointing at.

Timing gives you room. On Amazon EKS, the newest Kubernetes version listed under standard support is 1.36, so the managed path to 1.37 is not open yet. The cleanup, however, is yours regardless of who runs your control plane.

What to do this quarter

  1. Before any 1.36 to 1.37 upgrade, run kubectl api-resources --api-group=scheduling.k8s.io and then list and delete every remaining v1alpha2 Workload and PodGroup object. This is required whether or not you ever intend to use the feature.
  2. Grep your manifests and controllers for PodGroupTemplateRef, PodGroupScheduled, GangScheduling, and WorkloadAwarePreemption. Every hit is a rename or a removal.
  3. If you want the feature, confirm with your platform provider that you can set GenericWorkload and DRAWorkloadResourceClaims on both kube-apiserver and kube-scheduler. On a managed control plane that question has to be answered before the design work, not after.
  4. Once enabled, watch the new scheduler metrics queued_entities and queue_incoming_entities_total, which count scheduling entities — Pods or PodGroups — in and arriving at the queues. Threshold: if queued_entities holds above your steady-state Pod count for more than one training job duration, PodGroups are waiting rather than placing, and the gang is doing nothing but adding a queue.

The rest of the release is where the accelerator story quietly advanced anyway. DRA device taints and tolerations reached Stable in resource.k8s.io/v1, DRA extended resource support reached Stable, and ResourceClaim status data reached Stable so a driver can report per-device detail such as an assigned network address. All three are listed at v1.37 as GA with a default of true, and none of them needs a PodGroup. Getting the boring default-on half into production first is the sequencing from proof of concept to production argues for.

What would show this reading is wrong: open pkg/features/kube_features.go at the exact tag your distribution builds from and read the GenericWorkload entry. If it carries a default of true there, or a vendor patches it on, gang scheduling does arrive by upgrade after all and the only work left is the v1alpha2 cleanup. Check the tag, not the upstream release — that file is the one place the answer is unambiguous.

Ready to start?

Turn this into a plan for your team.

One week, fixed fee: a working session with your team, a prioritized use-case backlog, and an ROI model for the opportunities worth chasing.

Book an AI Opportunity Sprint