Cluster and workload management

As a Union.ai deployment grows past a single cluster, you need to control where a workload runs and under what limits. Three primitives do this:

• Cluster pool: an isolation boundary. The clusters and queues inside a pool share one data plane: the same object store, secret store, and container registry. Work cannot cross from one pool to another (see Crossing a pool boundary).
• Cluster: an execution cluster that lives in exactly one pool.
• Queue: what you submit work to. A queue lives in one pool, routes work to one or more clusters in that pool, and applies the concurrency, depth, priority, and fairness limits for the work it admits.

Tooling

Pools, clusters, and queues are managed with the flyte CLI or the flyteplugins.union.remote Python objects, and are set up by your platform administrator. These are administrative tasks; most workflow authors only need task-side queue routing.

Standing up a self-managed cluster?

The pages here manage the control-plane records for pools, clusters, and queues. They do not provision the data plane itself: the cloud resources (object store, secret store, registry) and the Helm release that registers a cluster with the control plane.

If you run a self-managed deployment, provision the data plane first with Self-managed deployment (for example, Data plane setup on AWS), then use the commands here to manage the pool, cluster, and queue records that route work to it.

How they fit together

            flowchart TD
    Org["Organization"]
    Org --> PD
    Org --> PP

    subgraph PD["Cluster pool: default"]
        direction TB
        QD["Queue: default<br/>(any project/domain)"]
        CA["Cluster A"]
        CB["Cluster B"]
        QD --> CA
        QD --> CB
    end

    subgraph PP["Cluster pool: prod"]
        direction TB
        QP["Queue: prod-queue<br/>(myproj / production)"]
        CC["Cluster C"]
        CD["Cluster D"]
        QP --> CC
        QP --> CD
    end
        

A cluster pool is an isolation boundary: both clusters and queues live inside a pool, and everything in it shares one data plane. A queue routes work to one or more clusters in its own pool. The platform picks any healthy cluster in the pool (or the specific clusters you pin it to).

The key invariant: a queue can never reach a cluster outside its pool, because a run’s inputs, code, and secrets are uploaded to that pool’s data plane and no other pool’s clusters can read them. That is what makes a pool an isolation boundary.

Crossing a pool boundary

Because pools don’t share a data plane, they don’t connect. You cannot move a run, a queue, or a cluster from one pool to another in place. Crossing a pool boundary means physically re-landing the workload in the destination pool’s data plane: moving its data, containers (images), code, and secrets into the new pool’s object store, registry, and secret store. This is deliberate friction: it keeps in-flight work from ever pointing at storage it can’t read, and it’s why pool changes are rare and explicit (moving work between pools is a drain-and-replace migration).

In this section