Image Builder

Union Image Builder supports the ability to build container images within the dataplane. This enables the use of the remote builder type for any defined Container Image.

Configure the use of remote image builder:

flyte create config --builder=remote --endpoint...

Write custom container images:

env = flyte.TaskEnvironment(
    name="hello_v2",
    image=flyte.Image.from_debian_base()
        .with_pip_packages("<package 1>", "<package 2>")
)

By default, Image Builder is disabled and has to be enabled by configuring the builder type to remote in flyte config

Requirements

  • The image building process runs in the target run’s project and domain. Any image push secrets needed to push images to the registry will need to be accessible from the project & domain where the build happens.

Configuration

Image Builder is configured directly through Helm values.

imageBuilder:

  # Enable Image Builder
  enabled: true

  # -- The config map build-image container task attempts to reference.
  # -- Should not change unless coordinated with Union technical support.
  targetConfigMapName: "build-image-config"

  # -- The URI of the buildkitd service. Used for externally managed buildkitd services.
  # -- Leaving empty and setting imageBuilder.buildkit.enabled to true will create a buildkitd service and configure the Uri appropriately.
  # -- E.g. "tcp://buildkitd.buildkit.svc.cluster.local:1234"
  buildkitUri: ""

  # -- The default repository to publish images to when "registry" is not specified in ImageSpec.
  # -- Note, the build-image task will fail unless "registry" is specified or a default repository is provided.
  defaultRepository: ""

  # -- How build-image task and operator proxy will attempt to authenticate against the default #    repository.
  # -- Supported values are "noop", "google", "aws", "azure"
  # -- "noop" no authentication is attempted
  # -- "google" uses docker-credential-gcr to authenticate to the default registry
  # -- "aws" uses docker-credential-ecr-login to authenticate to the default registry
  # -- "azure" uses az acr login to authenticate to the default registry. Requires Azure Workload Identity to be enabled.
  authenticationType: "noop"

  buildkit:

    # -- Enable buildkit service within this release.
    enabled: true

    # Configuring Union managed buildkitd Kubernetes resources.
    ...

Authentication

AWS

By default, Union is intended to be configured to use IAM roles for service accounts (IRSA) for authentication. Setting authenticationType to aws configures Union image builder related services to use AWS default credential chain. Additionally, Union image builder uses docker-credential-ecr-login to authenticate to the ecr repository configured with defaultRepository.

defaultRepository should be the fully qualified ECR repository name, e.g. <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<REPOSITORY_NAME>.

Therefore, it is necessary to configure the user role with the following permissions.

{
  "Effect": "Allow",
  "Action": [
    "ecr:GetAuthorizationToken"
  ],
  "Resource": "*"
},
{
  "Effect": "Allow",
  "Action": [
    "ecr:BatchCheckLayerAvailability",
    "ecr:BatchGetImage",
    "ecr:GetDownloadUrlForLayer"
  ],
  "Resource": "*"
  // Or
  // "Resource": "arn:aws:ecr:<AWS_REGION>:<AWS_ACCOUNT_ID>:repository/<REPOSITORY>"
}

Similarly, the operator-proxy requires the following permissions

{
  "Effect": "Allow",
  "Action": [
    "ecr:GetAuthorizationToken"
  ],
  "Resource": "*"
},
{
  "Effect": "Allow",
  "Action": [
    "ecr:DescribeImages"
  ],
  "Resource": "arn:aws:ecr:<AWS_REGION>:<AWS_ACCOUNT_ID>:repository/<REPOSITORY>"
}

AWS Cross Account access

Access to repositories that do not exist in the same AWS account as the data plane requires additional ECR resource-based permissions. An ECR policy like the following is required if the configured defaultRepository or ImageSpec’s registry exists in an AWS account different from the dataplane’s.

{
  "Statement": [
    {
      "Sid": "AllowPull",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<DATAPLANE_AWS_ACCOUNT>:role/<user-role>",
          "arn:aws:iam::<DATAPLANE_AWS_ACCOUNT>:role/<node-role>",
          // ... Additional roles that require image pulls
        ]
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ]
    },
    {
      "Sid": "AllowDescribeImages",
      "Action": [
        "ecr:DescribeImages"
      ],
      "Principal": {
        "AWS": [
          "arn:aws:iam::<DATAPLANE_AWS_ACCOUNT>:role/<operator-proxy-role>",
        ]
      },
      "Effect": "Allow"
    },
    {
      "Sid": "ManageRepositoryContents"
      // ...
    }
  ],
  "Version": "2012-10-17"
}

In order to support a private ImageSpec base_image the following permissions are required.

{
  "Statement": [
    {
      "Sid": "AllowPull",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<DATAPLANE_AWS_ACCOUNT>:role/<user-role>",
          "arn:aws:iam::<DATAPLANE_AWS_ACCOUNT>:role/<node-role>",
          // ... Additional roles that require image pulls
        ]
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ]
    },
  ]
}

Google Cloud Platform

By default, GCP uses Kubernetes Service Accounts to GCP IAM for authentication. Setting authenticationType to google configures Union image builder related services to use GCP default credential chain. Additionally, Union image builder uses docker-credential-gcr to authenticate to the Google artifact registries referenced by defaultRepository.

defaultRepository should be the full name to the repository in combination with an optional image name prefix. <GCP_LOCATION>-docker.pkg.dev/<GCP_PROJECT_ID>/<REPOSITORY_NAME>/<IMAGE_PREFIX>.

It is necessary to configure the GCP user service account with iam.serviceAccounts.signBlob project level permissions.

GCP Cross Project access

Access to registries that do not exist in the same GCP project as the data plane requires additional GCP permissions.

  • Configure the user “role” service account with the Artifact Registry Writer.
  • Configure the GCP worker node and union-operator-proxy service accounts with the Artifact Registry Reader role.

Azure

By default, Union is designed to use Azure Workload Identity Federation for authentication using user-assigned managed identities in place of AWS IAM roles.

  • Configure the user “role” user-assigned managed identity with the AcrPush role.
  • Configure the Azure kubelet identity ID and operator-proxy user-assigned managed identities with the AcrPull role.

Private registries

Follow guidance in this section to integrate Image Builder with private registries:

GitHub Container Registry

  1. Follow the GitHub guide to log in to the registry locally.
  2. Create a Union secret:
flyte create secret --type image_pull --from-docker-config --registries ghcr.io SECRET_NAME

This secret will be available to all projects and domains in your tenant. Learn more about Union Secrets Check alternative ways to create image pull secrets in the API reference

  1. Reference this secret in the Image object:
env = flyte.TaskEnvironment(
    name="hello_v2",
    # Allow image builder to pull and push from the private registry. `registry` field isn't required if it's configured
    # as the default registry in imagebuilder section in the helm chart values file.
    image=flyte.Image.from_debian_base(registry="<my registry url>", name="private", registry_secret="<YOUR_SECRET_NAME>")
        .with_pip_packages("<package 1>", "<package 2>"),
    # Mount the same secret to allow tasks to pull that image
    secrets=["<YOUR_SECRET_NAME>"]
)

This will enable Image Builder to push images and layers to a private GHCR. It’ll also allow pods for this task environment to pull this image at runtime.