Gang scheduling on Union applies to clustered tasks (`ClusteredTaskEnvironment`). This post covers what it guarantees, how it works, and how to use it.
The problem
A multi-node training job is a group of workers that have to start together. With PyTorch DDP, FSDP or JAX, every rank joins a rendezvous before the first step runs. If 7 of 8 workers are up and the 8th can't get a node, the 7 don't do partial work. They sit on their GPUs and wait.
The Kubernetes scheduler places pods one at a time, so a multi-node job can end up partly placed. Two things then go wrong:
- Idle GPUs. N−1 workers hold accelerators while the last one waits for a node that may never arrive. Nothing releases them until a human notices.
- Jobs blocking each other. Two 4-node jobs on a 6-node pool can each grab 3 nodes. Neither can finish starting.
Gang scheduling treats the whole group as one unit. Below is what Union does about each of them.
What you get
On Union, a clustered task runs as a Kubernetes JobSet, with one worker pod per node. With gang admission turned on, Kueue sits in front of that JobSet:
- No pods until the whole gang fits. The JobSet is created suspended, with zero pods and zero GPUs. Kueue reserves quota for every worker in one step, then lets the pods be created. A gang that doesn't fit holds nothing while it waits. This fixes problem 2: the second 4-node job waits with no pods instead of grabbing half a pool.
- A time limit on partial starts. If an admitted gang isn't fully Ready within 15 minutes, Kueue removes all its pods, frees the quota and requeues the gang with backoff. This fixes problem 1: GPUs held by a partial gang are given back after at most 15 minutes, with no human involved. A release doesn't use a JobSet restart or one of your retries.
- Accurate status reporting. The task shows `RUNNING` only when every worker is Ready. While held it shows `WAITING_FOR_RESOURCES`, and while workers come up it shows `INITIALIZING` with a live count.
"All or nothing" here applies to admission, not placement. Once Kueue admits a gang, Kubernetes still places its pods one by one as nodes arrive. The 15-minute window limits how long a partly placed gang can exist.
Two levels: Union decides what, Kueue decides when
Union's control plane already schedules. Its queue decides which task runs, in what order, on which cluster, within queue quotas. A gang that can never fit is rejected there before it reaches the cluster: a 3-worker job on a node pool capped at 2 nodes fails immediately as infeasible.
The control plane can't see what happens inside the cluster after it dispatches a gang: a node that fails to launch, or pods placed one at a time. Kueue covers that part, in each cluster.
Kueue's quota is set to each node pool's ceiling (instance size × max nodes), the same limits the control plane uses, so the two never disagree about whether a gang can fit. Ordering between gangs comes from Union's queue. In our tests, two gangs that each needed a whole pool were dispatched one after the other by Union's queue, and Kueue never had to hold the second.
Using it
Gang admission is opt-in per environment: add the queue label to the pod template. Clustered tasks without the label run exactly as they did before.
Requires `flyte` SDK 2.11.2 or later.
Where the queue comes from:
- On Union, we set up Kueue and a `shared` queue for your tenant, covering your cluster's CPU, GPU and accelerator node pools. Ask us to enable it.
- On open-source Flyte, install Kueue yourself, create your own queue, and put its name in the label. The plugin works with any queue name. One difference: `max_queued_time` and `deadline` are enforced on Union but not yet in open-source Flyte.
If you name a queue that doesn't exist, the task fails within seconds with `KueueLocalQueueNotFound` instead of waiting.
Always set `max_queued_time` and `deadline` for gangs. They're the only limits on how long a gang waits. `max_queued_time` covers one attempt, until every worker is Ready, and starts over when a retry relaunches the gang. `deadline` starts when the run is first queued and holds across every retry and restart, so it's the hard limit on the whole run.
What a run looks like
A real 2-node L4 run, starting with no GPU nodes in the cluster:
Submission to `RUNNING` took about 2 minutes, well inside the 15-minute window.
The messages you'll see:
Because `RUNNING` now means every worker is Ready, `max_runtime` measures training time, not node provisioning.
Two jobs, not enough GPUs for both
We submitted two 3-worker training jobs, `team-a` and `team-b`, at the same moment to a node pool that can hold 5 L4 GPUs. `team-a` got all three GPUs and its workers started together. `team-b` held no pods and no GPUs while it waited 1m 52s, then all three of its workers started 5 seconds after `team-a` finished training.

Two 3-worker L4 training jobs submitted together to a 5-GPU pool. Each row is one worker.
How to read the chart:
- Pale segment: the worker's container has started; it's starting up and joining the rest of its gang. It holds its GPU but does no training yet.
- Solid segment: every worker in the gang has joined, and the whole gang is training.
- Blue is `team-a`, orange is `team-b`. The time axis starts when both jobs were submitted.
Within each gang, the workers start within a second of each other and begin training at the same moment, and no `team-b` worker starts until every `team-a` worker has finished.
Retries: what counts against you
A clustered task can start over at three levels. Only one uses your `retries`:
- JobSet restarts (`failure_policy.max_restarts`). A worker crashed, so every pod in the gang is recreated in place. The task stays `RUNNING`. No retry is used.
- User retries. Your code failed after its restarts ran out. A new attempt starts with a new JobSet. This uses one of `retries`, after your backoff.
- System retries. Infrastructure problems: a stopped queue, failed nodes, Kueue giving up after repeated requeues. No retry is used.
Waiting is never a retry. A held gang, or one Kueue released before it fully started, keeps waiting on the same attempt. Failure messages name the cause and, for user retries, how many you have left. Whatever happens, `deadline` keeps counting across all of these.
Under the hood
- Kueue configuration: manages only JobSets that carry a queue label (`manageJobsWithoutQueueName: false`). `waitForPodsReady` is 15 minutes, with requeue backoff starting at 60 seconds, capped at 30 minutes, and at most 5 requeues. Queueing is `BestEffortFIFO`. Kueue checks quota for CPU, memory, GPUs and accelerators. Other resources a pod asks for, such as local disk or EFA network interfaces, are still enforced by Kubernetes when pods are placed, but they don't hold up admission.
- Open-source Flyte: the clustered plugin changes are upstream in Flyte. A suspended JobSet is never reported as running, `RUNNING` requires every worker Ready, evictions are classified as user or system retries from Kueue's own reason, and a missing queue fails fast.
- Union: installs and runs Kueue in each cluster, and creates and maintains the queues and quotas from the node pool definitions.
If you run multi-node training on Union and want to try it, let us know. It's one label.
Find a complete example here.







