Running in a local cluster

Running in a local Kubernetes cluster

Ultimately you will be running your workflows in a Kubernetes cluster in Union.ai. But it can be handy to try out a workflow in a cluster on your local machine.

First, ensure that you have Docker (or a similar OCI-compliant container engine) installed locally and that the daemon is running.

Then start the demo cluster using uctl:

$ uctl demo start

Configuration

When uctl starts the cluster in your local container engine it also writes configuration information to the directory ~/.union/.

Most importantly, it creates the file ~/.union/config-sandbox.yaml. This file holds (among other things) the location of the Kubernetes cluster to which we will be deploying the workflow:

admin:
  endpoint: localhost:30080
  authType: Pkce
  insecure: true
console:
  endpoint: http://localhost:30080
logger:
  show-source: true
  level: 0

Right now this file indicates that the target cluster is your local Docker instance (localhost:30080), but later we will change it to point to your Union.ai cluster.

Later invocations of uctl or union will need to know the location of the target cluster. This can be provided in two ways:

  1. Explicitly passing the location of the config file on the command line
    • uctl --config ~/.union/config-sandbox.yaml <command>
    • union --config ~/.union/config-sandbox.yaml <command>
  2. Setting the environment variable UNION_CONFIGto the location of the config file:
    • export UNION_CONFIG=~/.union/config-sandbox.yaml

In this guide, we assume that you have set the UNION_CONFIG environment variable in your shell to the location of the configuration file.

Start the workflow

Now you can run your workflow in the local cluster simply by adding the --remote flag to your union command:

$ union run --remote \
          workflows/example.py \
          training_workflow \
          --hyperparameters '{"C": 0.1}'

The output supplies a URL to your workflow execution in the UI.

Inspect the results

Navigate to the URL produced by union run to see your workflow in the Union.ai UI.

Local cluster with default image 

$ union run --remote my_file.py my_workflow

Where union is configured to point to the local cluster started with uctl demo start.

  • Task code runs in the environment of the default image in your local cluster.
  • Python code is dynamically overlaid into the container at runtime.
  • Only supports Python code whose dependencies are installed in the default image (see here).
  • Includes a local S3.
  • Supports some plugins but not all.
  • Single workflow runs immediately.
  • Workflow is registered to a default project.
  • Useful for demos.

Local cluster with custom image 

$ union run --remote \
              --image my_cr.io/my_org/my_image:latest \
              my_file.py \
              my_workflow

Where union is configured to point to the local cluster started with uctl demo start.

  • Task code runs in the environment of your custom image (my_cr.io/my_org/my_image:latest) in your local cluster.
  • Python code is dynamically overlaid into the container at runtime
  • Supports any Python dependencies you wish, since you have full control of the image.
  • Includes a local S3.
  • Supports some plugins but not all.
  • Single workflow runs immediately.
  • Workflow is registered to a default project.
  • Useful for advanced testing during the development cycle.