Prepare infrastructure

This page walks you through creating the resources needed for a Union data plane on generic (on-premise) Kubernetes. If you already have these resources, skip to Deploy the dataplane.

Kubernetes Cluster

You need a Kubernetes cluster running one of the most recent three minor Kubernetes versions. See Cluster Recommendations for networking and node pool guidance.

If you don’t already have a cluster, common options for provisioning one include:

• kubeadm — the standard Kubernetes bootstrap tool
• k3s — lightweight Kubernetes distribution
• RKE2 — Rancher’s hardened Kubernetes distribution

Regardless of how you create your cluster, verify the following requirements are met:

• Kubernetes version: one of the most recent three minor versions. Learn more.
• Networking: a CNI plugin is installed and functioning (e.g. Calico, Flannel, Cilium).
• DNS: CoreDNS is running in the cluster.
• Storage: a default StorageClass is configured for persistent volume claims.
• Load balancer or ingress: an ingress controller or load balancer is available for exposing services.

Union supports autoscaling and the use of spot (interruptible) instances if your infrastructure provides them.

Object Storage

Each data plane uses S3-compatible object storage (such as MinIO) to store data used in workflow execution. Union recommends the use of two buckets:

1. Metadata bucket: contains workflow execution data such as task inputs and outputs.
2. Fast registration bucket: contains local code artifacts copied into the Flyte task container at runtime when using flyte deploy or flyte run --copy-style all.

You can also choose to use a single bucket.

Create the buckets. For example, using the MinIO Client (mc):

# Set an alias for your MinIO server (if not already configured)
mc alias set myminio https://minio.example.com MINIO_ACCESS_KEY MINIO_SECRET_KEY

# Create the buckets
mc mb myminio/union-metadata
mc mb myminio/union-fast-reg

CORS Configuration

To enable the Code Viewer in the Union UI, configure a CORS policy on your bucket(s). This allows the UI to securely fetch code bundles directly from storage.

Save the following as cors.json:

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://*.unionai.cloud"],
      "AllowedMethods": ["GET", "HEAD"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3600
    }
  ]
}

Apply it to both buckets:

mc anonymous set-json cors.json myminio/union-metadata
mc anonymous set-json cors.json myminio/union-fast-reg

Consult your object storage provider’s documentation for the equivalent configuration if you are not using MinIO.

Data Retention

Union recommends using lifecycle policies on these buckets to manage storage costs. See Data retention policy for more information.

Container Registry

You need a container registry accessible from your cluster for Image Builder to push and pull container images. Options include:

• A private Docker Registry
• Harbor
• Any OCI-compliant registry

For a basic private Docker Registry, you can start one with:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Note the registry URL (e.g. registry.example.com:5000/union) — you will configure it in your Helm values.

Identity & Access

On generic Kubernetes, Union authenticates to object storage and the container registry using static credentials (access key and secret key). These are configured in the generated values file during deployment.

Storage credentials

Ensure the credentials you provide have read/write access to your storage bucket(s). If you are using MinIO, you can create a dedicated access key pair through the MinIO Console or the mc CLI:

# Create a new access key on your MinIO server
mc admin user add myminio union-service GENERATED_SECRET_KEY

# Attach a read/write policy for the Union buckets
mc admin policy attach myminio readwrite --user union-service

Registry credentials

Ensure you have push/pull credentials for your container registry. The specifics depend on your registry choice (Docker Registry basic auth, Harbor robot accounts, etc.).