Originally published on kuryzhev.cloud
A distributed training job asks for eight GPU workers. The scheduler places six, and the last two stay Pending. The six running pods sit on expensive accelerators waiting for peers that may not arrive for an hour. Nothing is broken, and nothing is making progress either. PodGroup gang scheduling exists to end that kind of deadlock. In Kubernetes 1.37 it is in beta.
The v1.37 release announcement lists Workload/PodGroup gang scheduling as beta, alongside GA for DRA extended resources (GPUs). The feature existed as alpha in earlier releases. With the beta, "all eight GPU pods or none" becomes a more mature built-in scheduler capability, rather than something you bolt on with a second scheduler.
Short answer: if your jobs need N pods running together (MPI, PyTorch distributed, Ray worker sets), do three things:
- Declare a Workload with a gang policy and a
minCount. - Point the pods at it.
- Verify the feature gates, API enablement and API version on your cluster first, because beta field names are version-dependent.
Skip it for independent replicas and single-pod jobs. Also skip it on clusters where Volcano, or Kueue's all-or-nothing admission, already solves partial placement for you.
What this actually does
Classic scheduling is pod by pod. The scheduler picks a node for pod A, binds it, then moves on to pod B. For a job that is useless unless all members run, this creates partial placement: some pods hold resources while others wait.
Gang scheduling changes the unit:
- Pods belong to a group declared through the Workload API.
- The group has a minimum member count.
- The scheduler only binds the group once that many pods can be placed.
If the group cannot be placed, none of its pods bind, and the GPUs stay free for other work.
Be clear about what it does not do. It is an all-or-nothing placement mechanism, not a queue:
- It does not decide which job goes first.
- It does not enforce team quotas.
- It does not preempt other tenants fairly.
Those remain jobs for admission controllers such as Kueue. It also does not, by itself, guarantee that the pods land close together on the network. Topology-aware placement is a separate concern, so check the scheduling docs for your version.
How people use it wrong
The most common misuse is treating the gang as a quota system. Teams enable it, submit twenty 8-pod jobs against a cluster that fits three, and expect orderly turn-taking. What they get is seventeen groups competing in the scheduler queue, with no fairness policy beyond what the scheduler provides. That is an admission problem, not a placement problem.
Watch out for a mismatched minCount. If a Job has parallelism: 8 but the gang says minCount: 4, the group can start with half the workers. Your framework then blocks on a rendezvous for the missing ranks, which is the original deadlock with extra steps. Set minCount to what the framework needs to initialize, not what is convenient.
Another trap is assuming the feature is on and the API shape is fixed. Alpha-era documentation used one field layout, and the 1.37 beta may have moved things, including how PodGroup objects relate to the Workload. Copying a manifest from an older blog post is the quickest way to get a silent no-op: pods schedule individually because nothing references the group correctly.
Finally, people forget the cost of waiting. A gang that can never fit, for example eight GPUs requested on a pool of six, stays Pending forever. Without an alert, nobody notices until a researcher asks why the run never started.
The correct approach
Start by confirming what your control plane actually serves. Run these before writing any YAML:
# Which gang/workload API versions does this cluster serve? kubectl api-resources | grep -i -E 'workload|podgroup' kubectl api-versions | grep -i scheduling # kube-apiserver feature gates only (names are version-dependent, so grep broadly). # kube-scheduler gates must be checked on the scheduler's own metrics endpoint or flags. kubectl get --raw /metrics | grep kubernetes_feature_enabled | grep -i -E 'gang|workload' # Authoritative field names for YOUR version kubectl explain workload.spec --recursive | head -40 🧪 Lab — try this in your browser
- Kubernetes Manifest Linter — Lint multi-document Kubernetes YAML for production readiness and security, with line:col diagnostics, severity grouping, a 0-100 score and fix hints.
Beta APIs are typically not served by default. If nothing shows up, the Workload API may need to be enabled on kube-apiserver with --runtime-config. The matching feature gates may also need to be set on both kube-apiserver and kube-scheduler. On managed clusters, check what your provider allows.
Then declare the group and reference it from the pods. The manifest below shows the general shape. Treat the apiVersion and exact field names as placeholders, and reconcile them with the kubectl explain output above and the Kubernetes scheduling documentation.
# Illustrative shape only. Confirm apiVersion and fields with kubectl explain. apiVersion: scheduling.k8s.io/v1beta1 # check: may differ in 1.37.x kind: Workload metadata: name: train-llm namespace: ml spec: podGroups: - name: workers policy: gang: minCount: 8 # must equal the ranks your framework needs --- apiVersion: batch/v1 kind: Job metadata: name: train-llm namespace: ml spec: completionMode: Indexed parallelism: 8 completions: 8 template: spec: restartPolicy: Never workloadRef: # links each pod to the group name: train-llm podGroup: workers # verify field name with kubectl explain pod.spec.workloadRef containers: - name: trainer image: registry.example.com/ml/trainer:2026.11 resources: limits: nvidia.com/gpu: 1 # device plugin or a DRA driver via extended resources, see below To verify it worked, submit the Job on a cluster with fewer than eight free GPUs.
- Expected result: all eight pods stay Pending together, with scheduling events referring to the group, not a mix of Running and Pending pods.
- After you free capacity: the pods should move to Running together.
- If some pods run while others wait: the reference is not being honored. Check
kubectl describe podfor the workload reference first.
Advanced patterns
Pair it with Kueue for admission. Let Kueue decide whether and when a job is admitted against team quotas, and let the gang guarantee atomic placement once admitted. This split avoids the "seventeen groups fighting" problem. It costs another controller to operate, so it mainly pays off when multiple teams share GPUs. Clusters that already rely on Kueue's own all-or-nothing admission can evaluate whether the scheduler-level gang adds anything for them.
Use DRA for the devices. With DRA extended resources GA in 1.37, a DRA driver can satisfy an existing extended-resource request such as nvidia.com/gpu, through a DeviceClass mapped to that resource name. You do not have to rewrite pods to use explicit ResourceClaims. Pods that need richer device selection can still use ResourceClaims directly. Combined with a gang, the "all or nothing" semantics then cover the devices too. Driver support varies, so verify in your driver's documentation how device allocation interacts with group placement.
Multiple groups per workload. A job with one launcher and many workers can declare separate groups with different minimums, so worker-count rules do not block the launcher. Some job operators (for example JobSet or training operators) may generate Workload objects for you in newer versions. Check whether yours already does before hand-writing them.
Watch out for autoscaling. Node autoscalers work from Pending pods. Whether yours understands that a gang needs all eight nodes at once, rather than scaling one at a time, depends on the autoscaler and its version. Read its release notes before relying on scale-from-zero GPU pools.
Performance notes
Gang scheduling trades latency for utilization. Group placement is a larger decision than single-pod placement, so scheduling a group takes more work than scheduling one pod, and a group that does not fit gets retried. No official numbers are cited here. Measure on your own cluster with the scheduler's metrics, for example scheduler_pending_pods and scheduler_pod_scheduling_sli_duration_seconds, and compare before and after enabling the gang.
The alert below uses kube_pod_status_phase from kube-state-metrics. It fires on any pod Pending in the namespace, not only on gang members, so keep training jobs in a dedicated namespace.
# Prometheus alert: pods in the training namespace Pending for 30 minutes groups: - name: gang-scheduling rules: - alert: TrainingPodsStuckPending expr: sum by (namespace) (kube_pod_status_phase{phase="Pending", namespace="ml"}) > 0 for: 30m labels: severity: ticket annotations: summary: "Pods in {{ "{{" }} $labels.namespace }} Pending 30m. Check gang minCount vs free GPUs" The real payoff is GPU-hours not burned on half-started jobs. The costs are real too:
- A bigger blast radius for one misconfigured
minCount. - A beta API that can still change between minor releases.
- Head-of-line effects, where a large gang waits while smaller jobs could have run.
Check the scheduler's behavior for your version, and consider keeping small jobs in a separate namespace or priority class.
When not to use it:
- Stateless inference Deployments.
- Jobs that tolerate elastic membership.
- Clusters with ample spare capacity, where partial placement resolves itself.
For beta features, pin a staging cluster on 1.37.x first and read the upgrade notes. More platform-engineering write-ups live on kuryzhev.cloud.

