Resource-aware scheduling: Union now knows what your clusters can run

Anyone who has run GPU workloads on Kubernetes knows the `Pending` pod status well. Sometimes it's waiting because the cluster is full and a node will free up in ten minutes. Sometimes it's waiting because it asked for an H100 on a cluster that only has A100s, and it will wait forever. From the outside those two pods look exactly the same.

Union now ships with resource-aware scheduling, which tells those two cases apart before anything reaches Kubernetes. The scheduler knows the node pools, node sizes, GPU types and maximum scale of every cluster a queue can send work to, so it only dispatches work that has somewhere to run. It's been live across Union's production control plane in all four regions since October 22.

What's resource-aware scheduling?

The scheduler checks whether an action can run on a cluster before it sends it there. To do that it tracks two things per cluster:

• Capacity: the cluster's node pools, the CPU, memory, ephemeral storage and accelerators on each pool's nodes, each pool's labels, whether it's spot or on-demand, and how many nodes each pool can scale to.
• Usage: what's already been scheduled onto that cluster and hasn't finished, so the scheduler knows how much room is left at full scale.

For every waiting action, oldest first and higher-priority queues first, it runs three checks in order:

1. Queue budget. Does the request fit in what's left of the queue's caps (CPU, memory, GPUs per type) and its concurrency limits?
2. Cluster room. Is there a cluster in the queue's selector whose capacity is at full scale (minus what's already scheduled there) that covers the whole request?
3. Node fit. On that cluster, does every container fit on a single node of some pool, with the right accelerator, a node selector that matches the pool's labels, and a spot pool if the task is interruptible?

If all three pass, the action goes to the least-loaded cluster that passed. If it could fit but there's no room right now, it stays `Queued`. If it could never fit, it fails as `INFEASIBLE`.

1. Work waits in the queue

When a cluster is full, work waits in the queue, which is where your scheduling policy lives: priority across queues, the scheduling policy within a queue, `max_queued_time`, and the console's view of what's waiting and why. All of that keeps applying until the moment the scheduler dispatches the action to a cluster with room for it.

The first week in production gave a good example. On October 23, a customer's production cluster that can run 305 T4s received a burst of about 3,500 GPU tasks. The scheduler filled every T4, kept the cluster at full use while the burst drained, and held everything else in the queue, in order, dispatching each task as a GPU freed up.

2. Impossible requests fail right away

When a request can never run, the scheduler fails it immediately, with the error code `INFEASIBLE` and no retry, so you find out in seconds. That happens when:

• the task asks for an accelerator none of the queue's clusters has,
• one of its containers is bigger than any node those clusters can launch,
• its total request is more than those clusters have at full scale,
• it's interruptible and no cluster has a spot pool it fits, or
• it asks for more than the queue's cap (this check applies with or without resource-aware scheduling).

The error message explicitly says what the action asked for and what the queue or the clusters offer, so you can usually tell right away whether to reduce the request, add a node pool, route to a different queue, or raise the cap.

Here's an example. The `research` queue routes to two clusters. Cluster A has an H100 pool with 8 GPUs per node that can scale to 4 nodes, so 32 H100s at full scale, and 28 are in use. Cluster B only has A100s.

3. GPUs are counted per type, and work goes to the least-loaded cluster

Because the scheduler knows which accelerators each pool has, it counts GPUs per accelerator type. An H100 request is charged against H100 capacity, and a T4 request against T4 capacity. When a task asks for a GPU without a type (`gpu=1`), the scheduler picks a device for it, and if that device type has a cap on the queue, the request counts against that cap too.

When a queue spans several clusters, each action goes to the least-loaded cluster that passes all three checks, and the Kubernetes scheduler inside that cluster picks the node.

What you write

Your task code stays the same. The scheduler reads the resources you already declare in each `TaskEnvironment`:

import flyte

env = flyte.TaskEnvironment(
    name="training",
    queue="research",
    resources=flyte.Resources(cpu=8, memory="64Gi", gpu="H100:1"),
)

On the platform side, each queue can have resource caps and a scheduling policy. A cap limits the total CPU, memory and GPUs (per type) that a queue's scheduled work may request at once, across all of its clusters:

flyte create queue research \
  --cluster-pool prod \
  --run-concurrency 100 \
  --action-concurrency 1000 \
  --max-resources cpu=512 --max-resources memory=4Ti \
  --max-accelerators H100=48 \
  --scheduling greedy_capacity

The scheduling policy decides what a queue does when the next action in line doesn't fit, whether that's because of the queue's caps or because no cluster has room. `greedy_capacity` skips it and schedules whatever fits behind it, which keeps GPUs busy and is the right choice for most queues. `strict_fifo` stops and waits, so freed capacity accumulates until the head of the line fits, which is what you want for a gang-style job that needs 32 GPUs at once. Because "doesn't fit" includes "the cluster is full," a `strict_fifo` queue can reserve GPUs for a large distributed training run as they free up.

To watch a queue live, including whether a strict-FIFO queue's head is blocked:

flyte get queue research --watch

Since waiting is now a scheduler state, you can also put a limit on it. `max_queued_time` fails an attempt that hasn't started within a time limit, and a parent task can catch the error and try a different GPU type or queue:

from datetime import timedelta

import flyte
import flyte.errors

gpu_env = flyte.TaskEnvironment(
    name="trainer",
    resources=flyte.Resources(cpu=8, memory="64Gi", gpu="H100:1"),
)
driver = flyte.TaskEnvironment(name="driver", depends_on=[gpu_env])

@gpu_env.task(timeout=flyte.Timeout(max_queued_time=timedelta(minutes=10)))
async def train(epochs: int) -> str:
    ...

# (queue, GPU) pairs, tried in order
PLACEMENTS = [("gpu-h100", "H100:1"), ("gpu-a100", "A100 80G:1")]

@driver.task
async def main(epochs: int = 3) -> str:
    for queue, gpu in PLACEMENTS:
        try:
            resources = flyte.Resources(cpu=8, memory="64Gi", gpu=gpu)
            return await train.override(queue=queue, resources=resources)(epochs=epochs)
        except flyte.errors.MaxQueuedTimeExceededError:
            print(f"Queue {queue} had no {gpu} capacity, moving on")
    raise RuntimeError("No queue had capacity")

So when something doesn't start, you know which case you're in. An `INFEASIBLE` error means you change the request or add a node pool, and a long wait means the cluster is full, which you can wait out or route around with `max_queued_time`.

How we rolled it out

We wanted a change to the scheduler under every production workload to be uneventful. So while resource-aware scheduling makes the decisions, the previous scheduler runs alongside it on the same inputs without acting on them, and the team can compare, for each org, what each one dispatched, held, or failed. The first week in production ran cleanly. For more on how the scheduler works inside, see Inside Union: Resource-Aware Scheduling for Flyte 2.0.

Scope and what's next

A few details are worth knowing when you plan around it:

• The scheduler plans against each pool's configured maximum size. If the cloud can't provide the nodes (an EC2 capacity error, a quota limit, an exhausted zone), a dispatched pod can still sit `Pending`, and `max_queued_time` is the guardrail for that. Scheduling against observed capacity is next on the roadmap.
• On Union BYOC, the scheduler learns each cluster's node pools automatically. On self-managed deployments, the Union team sets them up with you. A cluster whose node pools Union doesn't know yet is scheduled as if it had unlimited capacity.
• Union picks the cluster, and the Kubernetes scheduler inside it picks the node.
• Priority decides the order work starts in and never interrupts running work. Caps are ceilings rather than reservations, so a higher-priority queue can take capacity that a job on a lower-priority queue is waiting for.
• Backfill is in progress. It will let short work use capacity that a `strict_fifo` queue is holding for the head of the line, whenever doing so won't delay that job.
• The console shows caps, usage and blocked queue heads today, and a view of the node pools the scheduler is reasoning about is planned.

Recap

• Full clusters: work waits in the queue, under priority and scheduling policy, and starts as soon as there's room.
• Impossible requests: fail right away as `INFEASIBLE`, with a message that names the request and what's available.
• Multi-cluster queues: GPUs are counted per type, and each action goes to the least-loaded cluster that can run it.
• Caps and policy: per-queue CPU, memory and GPU caps, plus `greedy_capacity` or `strict_fifo`, now apply against what your clusters can actually hold.

Try it

To set caps and a scheduling policy on your queues, install the queue tooling and point a queue at your GPUs:

pip install "flyteplugins-union>=0.16.0"
flyte update queue research --max-accelerators H100=48 --scheduling greedy_capacity
flyte get queue --watch

The resource caps and scheduling docs cover caps, policies, and the full list of infeasible conditions. If your clusters are self-managed, or you're not sure Union has your node pools on file, talk to the Union team.