Kubeflow KFP vs. Union AI

Compare running a Bayesian sweep on Union AI with Katib / KFP. Toggle the Kubeflow card to switch Katib HPO with KFP ParallelFor execution.

From notebook to promoted model

Kubeflow KFP

Union AI

Steps to launch

Config to hand-write

Docker + YAMLbuild · push · kubectl

No K8s YAMLHydra config, typed Python

To run the sweep

Multi-step opsapply · watch · debug

1 command--multirun

Guided search

Tree of Parzen Estimatorsvia Katib service

Tree of Parzen EstimatorsOptuna via Hydra Sweeper

Kubeflow + Katib

algorithm: bayesianoptimization | tpe (suggestion service)

Pull the training loop out of the notebook

Refactor into a deterministic train.py: hyperparameters become CLI flags, and metrics must be emitted where a collector can parse them — printed as accuracy=<value> for the default StdOut collector, or written for the File / TFEvent collectors.

train.pyargparsestdout metric

Containerize and publish an image

Write a Dockerfile, build a dependency-locked image, and push it to a registry — plus an imagePullSecret if that registry is private, so the cluster can pull it during trials.

DockerfileregistryimagePullSecret

Author the Experiment as Kubernetes YAML

Hand-write the objective, parameter space, algorithm, trial counts, metricsCollectorSpec, trialTemplate, and resource requests as a Katib custom resource.

Experiment CRtrialTemplateCPU / GPU

Apply and operate on the cluster

kubectl apply, then watch Trials, Jobs, and Pods. Debugging means kubectl logs / describe against image pulls, RBAC, quotas, and failed metric collection.

kubectlPods / JobsRBAC

Retrieve the best trial and promote

Read currentOptimalTrial from status, then stand up a KFP pipeline plus CI/CD to retrain on approved data and push the model to a registry.

currentOptimalTrialKFPCI/CD

search: no suggestion service — you build it

Package each stage as a KFP component

Declare train / evaluate as a @dsl.component with typed Input/Output artifacts and the metric return contract the selector reads. Light components can lean on packages_to_install, but real training dependencies mean a custom base_image to build and push.

@dsl.componentbase_imagebuild + push

Generate the candidate configs

A component emits the search space as an explicit grid or random sample — workable only for a small, fixed set of trials known up front.

generate_candidatesgrid / randomstatic list

Build the search controller

Removing Katib removes its suggestion service, so you embed Optuna (or Ray Tune) yourself. The catch isn't embedding it — it's that ParallelFor is compiled topology and needs candidates up front, so an adaptive ask → score → tell loop can't live inside the pipeline. You drive rounds from outside, or recompile per round.

embed Optunacompile-time topologycontroller lives outside

Author the sweep as a KFP DSL pipeline

dsl.ParallelFor(parallelism=N) fans out each round of trials; dsl.Collected returns their scores to your controller and selector. The DSL looks like Python but describes topology, not local execution.

dsl.ParallelFordsl.Collectedparallelism=N

Compile and submit to the pipeline runtime

Compile the DSL, submit it to the KFP runtime, and make sure Kubernetes quotas, GPUs, and autoscaling support the concurrency. Retries and cleanup are wired in via exit handlers.

kfp compilepipeline runtimequotas / GPU

Select, retrain, and register

choose_winner from the collected scores, retrain on approved data, and register the model in downstream steps — while still operating the underlying Kubeflow services throughout.

choose_winnerretrainmodel registry

Union AI + Hydra + Optuna

hydra/sweeper=optuna · FlyteLauncher (remote) · n_trials × n_jobs

Define the experiment as composable Hydra config

flyteplugins-hydra accepts YAML or structured configs and passes the composed DictConfig into your Flyte tasks. Hydra owns the complete, composable schema and stable defaults; compose config groups and set tunable ranges as CLI overrides — interval(1e-4,1e-1) for continuous, choice(adam,sgd) for categorical. Training is a typed Union task that returns a float, so there's no stdout contract or metric string to parse.

flyteplugins-hydrainterval() / choice()typed task → float

Pick a grid or an adaptive Optuna search

Comma-separated overrides expand to a Cartesian grid (Hydra's BasicSweeper); selecting hydra/sweeper=optuna switches on adaptive TPE with n_trials and n_jobs. Same framework either way — Optuna's sampler proposes each candidate from the objective and the sweeper owns the study, so you write no controller loop and parse no metrics.

grid or adaptivehydra/sweeper=optunan_trials · n_jobs

One command — --multirun with the Flyte launcher — and Union runs every trialbuild · package · provision · isolated Flyte runs · artifacts + metadata · retrain the winner

FlyteLauncher runs each trial as a Flyte execution

--multirun with hydra/launcher=flyte sends every trial to Union as its own remote run; wait=True resolves each run's typed output to the objective float. Each execution is isolated, cached, and captured as lineage, and the best config feeds straight into the retraining workflow.

--multirunFlyteLauncher remotetrial = Flyte run

One command — the whole sweep Read the Docs

flyte hydra run --multirun --config-path conf --config-name training \
  train.py pipeline \
  --hydra-override hydra/sweeper=optuna \
  --hydra-override hydra.sweeper.n_trials=20 \
  --hydra-override hydra.sweeper.n_jobs=4 \
  --cfg "optimizer.lr=interval(1e-4,1e-1)"

Optuna's TPE sampler drives selection; FlyteLauncher runs each suggested, Hydra-composed config as its own remote Flyte execution.

Why the developer experience is better

No YAML, Dockerfiles, or kubectl. A typed-Python + CLI path from prototype to sweep.

Remote Image Building. The platform builds and pushes the container — nothing to install or configure locally.

Reusable containers. Opt in with reusable=ReusePolicy(...) and replicas persist across trials, so the sweep skips per-trial cold starts.

Live debug. flyte debug <run> relaunches a run with an in-pod SSH server and prints an ~/.ssh/config block for ssh or VS Code Remote-SSH. No ingress or port-forwarding. (Beta, via flyteplugins-union.)

Are you the hands-on type?

Spin up the whole sweep on your own GPUs in minutes — no cluster to stand up.

Get DevBox — run it locally