AWS infrastructure

This page walks you through creating the AWS resources needed for a Union data plane. If you already have these resources, skip to Deploy the dataplane.

Environment variables

Set these variables before running the commands below. Customize the names if you are deploying multiple data planes in the same AWS account.

export AWS_REGION=us-east-2                          # AWS region for all resources
export CLUSTER_NAME=union-dataplane                  # EKS cluster name
export BUCKET_PREFIX=union-dataplane                 # prefix for S3 buckets (must be globally unique)
export ECR_REPO_NAME=${ECR_REPO_NAME}    # ECR repository name
export IAM_ROLE_NAME=union-system-role               # IAM role name

EKS cluster

You need an EKS cluster running one of the most recent three minor Kubernetes versions. See Infrastructure recommendations for networking and node pool guidance.

If you don’t already have a cluster, create one with eksctl:

eksctl create cluster \
  --name ${CLUSTER_NAME} \
  --region us-east-2 \
  --version 1.31 \
  --node-type m5.2xlarge \
  --nodes 3 \
  --with-oidc \
  --managed

The following EKS add-ons are required and come pre-installed on managed clusters created with eksctl:

• CoreDNS
• Amazon VPC CNI
• Kube-proxy

If you created your cluster through other means, verify they are installed:

aws eks list-addons --cluster-name ${CLUSTER_NAME} --region ${AWS_REGION}

Union supports Autoscaling and the use of spot (interruptible) instances.

Networking and IP capacity

The AWS VPC CNI assigns one VPC IP per pod from the node’s subnet, so pod-IP exhaustion is the most common scale blocker. Every running pod holds an IP, and so does each node’s warm pool of spare IPs, so bursty, high-churn workloads compound subnet pressure. Size the VPC greedily up front — adding VPC CIDR blocks later works, but resizing existing subnets does not.

Suggested defaults for a production-scale data plane:

This sizing supports up to ~40,000 pods and ~9,000 nodes per VPC. When a single cluster scales past ~40,000 concurrent pods, add 10.1.0.0/16, 10.2.0.0/16, etc. as additional VPC CIDR blocks and provision new private subnets from them — the VPC CNI does not require contiguous CIDRs.

Relieving pod-IP pressure:

• Add CIDR blocks to the VPC (typically /16 per block). New private subnets become available immediately.
• Enable VPC CNI prefix delegation — allocates /28 prefixes (16 IPs) per ENI attachment instead of individual IPs.
• Trim the VPC CNI warm pool on large, bursty clusters (for example, big GPU array jobs that scale out many nodes at once). By default (WARM_ENI_TARGET=1) each node keeps a whole spare ENI’s worth of IPs free, which across thousands of nodes can leave tens of thousands of subnet IPs idle. Setting WARM_IP_TARGET together with MINIMUM_IP_TARGET in the VPC CNI add-on configuration holds a fixed number of free IPs per node instead (WARM_IP_TARGET overrides WARM_ENI_TARGET). The trade-off: the CNI calls the EC2 API much more often, and AWS warns that on large or high-churn clusters this can hit API throttling. With prefix delegation enabled, tune WARM_PREFIX_TARGET instead. See the VPC CNI configuration reference.

Union.ai deletes each task pod as soon as its task finishes, so completed pods do not hold IPs and there is no retention timer to tune.

Monitoring IP usage: scrape the VPC CNI metrics that the aws-node DaemonSet exposes on port 61678 (/metrics) to see subnet pressure before pods fail with FailedCreatePodSandBox. The key series are awscni_assigned_ip_addresses (IPs in use by pods), awscni_total_ip_addresses (all allocated IPs, including the warm pool), and awscni_no_available_ip_addresses (IP exhaustion events).

For the full set of per-cluster scaling ceilings (vCPU quotas, image pull rate, conntrack, etcd), see Scaling constraints.

S3

Each data plane uses S3 buckets to store data used in workflow execution. Union recommends the use of two S3 buckets:

1. Metadata bucket: contains workflow execution data such as task inputs and outputs.
2. Code bundle/Fast registration bucket: contains local code artifacts copied into the Flyte task container at runtime when using flyte deploy or flyte run --copy-style all.

You can also choose to use a single bucket.

Create the buckets:

aws s3api create-bucket \
  --bucket ${BUCKET_PREFIX}-metadata \
  --region ${AWS_REGION} \
  --create-bucket-configuration LocationConstraint=${AWS_REGION}

aws s3api create-bucket \
  --bucket ${BUCKET_PREFIX}-fast-reg \
  --region ${AWS_REGION} \
  --create-bucket-configuration LocationConstraint=${AWS_REGION}

CORS configuration

To enable the Code Viewer in the Union UI, configure a CORS policy on your buckets. This allows the UI to securely fetch code bundles directly from S3.

Save the following as cors.json:

{
  "CORSRules": [
    {
      "AllowedHeaders": ["*"],
      "AllowedMethods": ["GET", "HEAD"],
      "AllowedOrigins": ["https://*.unionai.cloud"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3600
    }
  ]
}

Apply it to both buckets:

aws s3api put-bucket-cors --bucket ${BUCKET_PREFIX}-metadata --cors-configuration file://cors.json
aws s3api put-bucket-cors --bucket ${BUCKET_PREFIX}-fast-reg --cors-configuration file://cors.json

Data retention

Union recommends using Lifecycle Policy on these buckets to manage storage costs. See Data retention policy for more information.

ECR

Create an ECR private repository for Image Builder to push and pull container images:

aws ecr create-repository \
  --repository-name ${ECR_REPO_NAME} \
  --region ${AWS_REGION} \
  --image-scanning-configuration scanOnPush=true

Note the repository URI from the output (e.g. <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/${ECR_REPO_NAME}). You will reference it when configuring IAM permissions below.

IAM

Create an IAM role that both the Union platform services and workflow task pods will use to access S3 and ECR. This role is assumed via IAM Roles for Service Accounts (IRSA).

1. Enable OIDC

If you created your cluster with --with-oidc above, this is already done. Otherwise, create an IAM OIDC provider for your EKS cluster:

eksctl utils associate-iam-oidc-provider --cluster ${CLUSTER_NAME} --region ${AWS_REGION} --approve

Get the OIDC provider URL (you’ll need it for the trust policy):

export OIDC_PROVIDER=$(aws eks describe-cluster \
  --region ${AWS_REGION} \
  --name ${CLUSTER_NAME} \
  --query "cluster.identity.oidc.issuer" \
  --output text | sed 's|https://||')

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

2. Create the IAM role

Save the following trust policy as trust-policy.json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "$OIDC_PROVIDER:aud": "sts.amazonaws.com"
                },
                "StringLike": {
                    "$OIDC_PROVIDER:sub": "system:serviceaccount:*"
                }
            }
        }
    ]
}

Substitute your values and create the role:

envsubst < trust-policy.json > /tmp/trust-policy.json

aws iam create-role \
  --role-name ${IAM_ROLE_NAME} \
  --assume-role-policy-document file:///tmp/trust-policy.json

3. Attach the S3 policy

Save as s3-policy.json (replace <BUCKET_PREFIX> with your actual prefix):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3BucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject*",
                "s3:GetObject*",
                "s3:ListBucket",
                "s3:PutObject*"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET_PREFIX>-metadata",
                "arn:aws:s3:::<BUCKET_PREFIX>-metadata/*",
                "arn:aws:s3:::<BUCKET_PREFIX>-fast-reg",
                "arn:aws:s3:::<BUCKET_PREFIX>-fast-reg/*"
            ]
        }
    ]
}

aws iam put-role-policy \
  --role-name ${IAM_ROLE_NAME} \
  --policy-name union-s3-access \
  --policy-document file://s3-policy.json

4. Attach the ECR policy

Save as ecr-policy.json (replace <AWS_REGION>, <AWS_ACCOUNT_ID>, and <REPOSITORY>):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ECRAuth",
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ECRReadWrite",
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer",
                "ecr:DescribeImages",
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload"
            ],
            "Resource": "arn:aws:ecr:<AWS_REGION>:<AWS_ACCOUNT_ID>:repository/<REPOSITORY>"
        }
    ]
}

aws iam put-role-policy \
  --role-name ${IAM_ROLE_NAME} \
  --policy-name union-ecr-access \
  --policy-document file://ecr-policy.json

5. Configure the service account annotation

In your Helm values, annotate the union-system service account with the role ARN:

commonServiceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/${IAM_ROLE_NAME}"

Deploy configuration

When you deploy the data plane, download the AWS values file and set the AWS-specific keys below. The shared global keys (UNION_CONTROL_PLANE_HOST, CLUSTER_NAME, ORG_NAME) are covered in the deploy walkthrough.

curl -O https://raw.githubusercontent.com/unionai/helm-charts/main/charts/dataplane/values.aws.yaml

Using the environment variables from above, set the following keys under global. The rest of the file (storage, service account annotations, IRSA) is templated from these values, so you do not need to edit it:

• Set global.AWS_ACCOUNT_ID to your AWS account ID. You can retrieve it with aws sts get-caller-identity --query Account --output text.
• Set global.AWS_REGION to ${AWS_REGION}.
• Set global.METADATA_BUCKET to ${BUCKET_PREFIX}-metadata.
• Set global.FAST_REGISTRATION_BUCKET to ${BUCKET_PREFIX}-fast-reg.
• Set global.BACKEND_IAM_ROLE_ARN to arn:aws:iam::${AWS_ACCOUNT_ID}:role/${IAM_ROLE_NAME} (where AWS_ACCOUNT_ID is your 12-digit account ID).
• Set global.WORKER_IAM_ROLE_ARN to the same value (or a separate role if you use distinct worker permissions).
• Optionally set imageBuilder.registryName to ${ECR_REPO_NAME} (defaults to union-dataplane; the chart auto-generates the full ECR URL from the account ID and region).