As AI teams scale, they run into the same wall: more workloads, more teams, and the same finite pool of compute underneath all of it. A backfill job kicks off at the wrong time. A training run claims every available GPU. Suddenly the live inference service that's supposed to answer in milliseconds is waiting in line behind a batch job nobody's watching closely.
This is what happens when shared infrastructure has no rules for who goes first. It's not a hardware problem. It's a scheduling problem, and it gets worse as organizations grow, not better.
Today we're launching Queues and the first set of Cluster Controls, two capabilities built to solve exactly this. Together they give platform teams a way to control how work gets scheduled across shared infrastructure, and visibility into what's actually happening on the clusters underneath it, so critical workloads keep running durably to completion even when everyone is competing for the same resources.
The problem: contention without control
Union.ai customers run a mix of workload types on the same infrastructure. Model training. Batch inference. Live, user-facing inference with real latency requirements. Ad hoc experimentation from a dozen different teams.
Without a mechanism to separate and prioritize these workloads, the loudest or largest job wins by default, regardless of what's actually urgent. Workloads that get starved for resources pose as serious a threat to durability as the logic failures Flyte already solves.
We built Queues to change that.
What’s launching today
Queues: control how work gets scheduled
A Queue is Union's mechanism for controlling scheduling: organizing runs and actions by priority, enforcing concurrency limits, ensuring fairness across teams, and routing workloads to the right infrastructure. Every queue routes to exactly one cluster pool, and the queue itself governs priority, fairness, concurrency, and depth within that pool.
Fairness across projects, out of the box.
By default, every project submitting to a queue is treated as a peer, regardless of how much volume it sends. The default algorithm cycles through projects one action at a time, so a high-volume team can't quietly starve everyone else. Where round-robin's bias toward earlier arrivals is a problem, a randomized-interleave mode is available instead. Fairness is automatic, and it's swappable per queue.
Independent concurrency limits.
Every queue can cap run concurrency (how many runs are active at once) and action concurrency (how many tasks are in flight) separately. That means a single backfill or high-traffic project can't consume all available capacity and starve the workloads sharing that infrastructure.
Priority tiers, evaluated per cluster pool.
Queues support strict Low, Medium, and High priority tiers. High-priority queues are always scheduled ahead of Medium and Low. Priority is intentionally non-preemptive: an arriving high-priority action won't interrupt a lower-priority action that's already running, so you get predictable ordering without unpredictable interruptions.
Targeted cluster routing.
A queue's cluster selector can target any healthy cluster in its pool, weighted equally, or a named subset. This lets teams pin specific workloads, like GPU-heavy jobs, to the infrastructure best suited for them. Selectors can be changed at any time with no drain required.
Runtime queue overrides. A single SDK call lets a task redirect itself to a different queue on the fly, as long as the target queue lives in the same cluster pool as the run.
How Queues fit into your org
None of this requires a redesign of how your teams already work. A few things make queues easy to adopt incrementally:
Scope a queue to the level that works for your team.
Queues can be global (available across the whole org), or you can specify which cluster pool and clusters within that pool a queue utilizes. A platform team might stand up one high-priority global queue for production inference, while individual teams leverage specific clusters for experimentation.
Set a depth limit to manage backlogs, not just live traffic.
Alongside run and action concurrency, every queue has a depth: the max number of items it will hold at once. This makes queues useful for backlog-style workloads too. A team can enqueue a large batch of backfill work, cap its depth, and let it drain at a controlled rate in the background, while a high-priority queue sharing the same cluster keeps moving ahead of it without being delayed.
Pick a queue right from the run launch form.
Queue selection isn't a separate admin step. When launching a run from the UI, a Queue selector sits right in the run's Settings panel, showing each option's cluster pool inline so you can see exactly which infrastructure class a run will land on before you launch it.
Live queue metrics.
Every queue exposes its own state: depth, current runs, and current actions in flight, both in the UI and via flyte get queue. Paired with the cluster-side workload view, this answers two questions at once: how busy is this queue, and which queues are busy on this cluster.
This is the first release, not the last. We'll keep shipping improvements to Queues on a regular cadence. Our next release is queue draining, the safe way to retire a queue without losing its historical references, landing in the next few weeks.
What’s next for durability under contention: Cluster Controls
Cluster Controls are the next layer of this same effort, rolling out in stages starting with an upcoming cluster release. Where Queues govern how work is scheduled, Cluster Controls govern how much visibility you have into the infrastructure that work runs on, turning every onboarded cluster into a fully described, continuously updated object instead of a black box.
A few things to expect in the first release: clusters register themselves automatically on install, going Healthy in the UI within a couple of minutes with no manual setup step. Platform admins get a live, browser-based view of every dataplane component's configuration, no SSH or YAML spelunking required. And the cluster detail page starts answering questions that today require guesswork, like which queue is actually consuming a cluster's capacity, how saturated the cluster's action-claim capacity is, and how much of that capacity Union's own components are using versus your workloads.
The criticality of durability under contention
Durability has always meant workflows recovering gracefully when something breaks. But it often ignores the idea that infrastructure dictates success as much as workflow logic. Durability should also mean resilience under contention.
Shared infrastructure only works if it holds up under contention, and contention is not an edge case. It's the normal state of any team running production AI workloads alongside experimentation, batch jobs, and training runs on the same clusters.
Queues and Cluster Controls exist to make that contention manageable. Priority and fairness settings mean critical runs, like live inference, don't get starved by a backfill nobody's watching. Cluster selectors mean workloads land on the infrastructure built for them. And as Cluster Controls rolls out in full, that same durability extends to visibility: platform teams will be able to see the health, configuration, and cost of every cluster they run, without waiting for something to break first.
The goal is simple. As your workloads and teams multiply, the infrastructure underneath them should stay predictable, not because nothing ever competes for it, but because you have the controls to manage that competition on your terms.
Get started
Queues are live today for Union.ai customers, with Cluster Controls following in the coming weeks. If you're evaluating Union.ai or want a walkthrough of how Queues fit into your current setup, book a chat with an engineer.
