Maximizing ML on Kubernetes
Where the time, the GPUs and the engineering hours actually go.
A field guide for teams running Kubeflow: where ML on Kubernetes loses time and GPU hours, and how to get them back. Covers the six subprojects plus KServe, the upgrade treadmill, cold-start GPU idle, multi-cluster capacity, and the agent and RL workloads a compiled DAG cannot express. With a TCO calculator and a GPU idle-tax model.
00 · Kubeflow setup
Your team is running seven distributed systems
Kubeflow presents as an ML platform. Operationally, it's a distribution: seven separate open-source projects, each with its own maintainers, release cadence, and failure modes, stitched together on Kubernetes.
Kubeflow Pipelines (KFP) orchestrates. Katib tunes. Trainer trains. Notebooks host development. Spark Operator handles data. Hub tracks models and metadata. And KServe serves, though the Kubeflow project no longer governs it: KServe sits in the Kubeflow Ecosystem, maintained by its own community in its own repo. Each is a genuine project doing genuine work, and each is a thing your platform team now integrates, secures, upgrades, and carries a pager for. The moment you adopt the distribution, you don't get a product. You get an internal product to run, assembled from parts you didn't write and can't stop maintaining.
“Free” describes the license. It does not describe the platform.
What's actually under the hood
Kubeflow's current maturity tables list six graduated subprojects (Pipelines, Katib, Trainer, Notebooks, Spark Operator and Hub) plus two incubating, Kale and the Kubeflow SDK. Serving is the seventh thing you run and the one the project no longer governs: KServe now sits in the Kubeflow Ecosystem, which Kubeflow defines as projects "not governed by the Kubeflow community" and supported by their own maintainers. Around all of it, the Kubeflow Community Distribution bundles Istio, cert-manager, Dex, OAuth2-proxy and Argo Workflows, plus Knative where KServe runs in its serverless mode. Each ships on its own governance and release cycle. Pipelines can be deployed on its own, and a team that only needs orchestration should: the weight below is the weight of the full distribution, which is what most teams adopt when they want the platform rather than just the DAG runner.
The community feels the weight of that directly. From the manifests project's own complexity thread: Kubeflow "is overly heavy-handed… the only app I can think of that virtually necessitates installation into its own dedicated Kubernetes cluster because of the assumptions it makes." Teams that already run their own ingress, cert management, or identity are forced to duplicate infrastructure or hand a whole cluster to Kubeflow alone.
01 · The overhead you operate
Seven release cycles, one upgrade treadmill
The seven projects don't coordinate their releases for you. So an upgrade isn't a version bump, it's a project.
Each component moves on its own schedule, with its own breaking changes, on top of a Kubernetes version and a dependency stack that also move. Reconciling all of it (testing that Pipelines still talks to KServe, that the new Dashboard manifests still authenticate through Dex, that Argo didn't shift under you) is recurring, skilled work that produces no models. It's the tax you pay just to stay in place. And it lands on your most expensive, hardest-to-hire people: the platform engineers you brought on to accelerate the team, now spending their weeks keeping the platform from drifting.
Why upgrades are never routine
- Kubeflow moved to calendar versioning (26.03.1, April 2026) with roughly two base releases a year, each folding in independently-versioned Pipelines, KServe, Katib, Trainer and more.
- Recent releases ship breaking structural changes (e.g. Dashboard manifest restructuring) and require coordinating Istio, cert-manager, and whatever else your install pulls in against a moving Kubernetes (1.36) target.
- Cloud-specific distributions hardcode additional dependencies on top, compounding the reconciliation for multi-cloud teams.
- There is no single "upgrade Kubeflow" button; there is a matrix of components you validate against each other, every cycle.
The license is $0. The upkeep is people. Estimate the standing cost of keeping Kubeflow running.
Kubeflow license: $0
$198,000
per year keeping it alive, the equivalent of 0.9 full-time engineers doing nothing but platform maintenance.
Annual upkeep cost = engineers × salary × time-share; FTE-equivalent = engineers × time-share. Illustrative: set inputs to your team.
02 · The GPU idle tax
The cost that hides in your cloud bill
The overhead is the cost you can see on the org chart. This one hides in your GPU spend, and the pipeline layer does nothing to stop it.
Kubeflow starts a fresh pod for every task, experiment, data pre-processing step and every HPO trial. If your pipeline launches thousands of short jobs, that means a cold start (image pull, container init, framework warm-up) on a GPU that is already allocated and billing by the second. And the orchestrator itself has nothing to claw it back: warm reuse and batching live in the serving layer you add on top (KServe, Ray Serve, vLLM, Triton), not in the pipeline that launched the task. So the accelerators you fought to get sit idle, cold-starting, on your dime, and closing the gap is another service to select, integrate, and operate. Union treats keeping GPUs hot as a platform responsibility, not a per-team hack, so the same work reclaims that time.
Where the time goes, and how Union recovers it
Kubeflow: new pod per task means a multi-GB CUDA image pull and init on each launch; between requests, a serving GPU idles unless you add a layer that packs the work.
Union (Flyte 2): Reusable Containers keep replicas live across actions via reusable=ReusePolicy(...), so after the first warm-up dispatch is <100ms; cold boot is cut from minutes to seconds; and DynamicBatcher / TokenBatcher pack requests by cost or tokens to keep the GPU saturated.
Not the same thing as a warm serving pool. Sub-100ms dispatch does exist elsewhere, in the serving layer: KServe with minReplicas ≥ 1, or Ray Serve actors. Both get there by holding capacity open, which means paying for GPUs that are up and waiting so that an inference request never meets a cold start. That is a reasonable trade for a latency-sensitive endpoint, and a different problem from this one. Here the work is a pipeline: the replica is live because actions are flowing through it, not because it is standing by for traffic. The gap is that Kubeflow's orchestrator has no equivalent at all, that means a Kubeflow pipeline pays for a fresh pod per task and the only way to avoid it is to move the work into a serving layer and start paying to hold it open.
Anchor on the two you know (GPUs running and price per GPU-hour), then set how hard the fleet churns.
Your GPU fleet
the two anchors + how hard it churns
Accelerators allocated to this workload and billing while up.
How often each GPU picks up a fresh task. Sweeps push this high.
Pod + multi-GB image pull + init. On Union, dispatch onto a live replica is <100ms.
The idle tax
GPU time you pay for, computing nothing
$25,000
burned per year on cold-start idle: GPUs running while pods pull images and initialize.
That's 1.1% of your $2.24M/yr GPU fleet, lost before a single epoch runs.
ReusePolicy; <100ms onto a live replica.03 · The scale ceiling
It works in the demo. Then the sweep gets real.
Kubeflow is fine at small scale. The trouble starts when your workload does what AI workloads do, fan out, and the bottleneck stops being your model and becomes the control plane itself.
A real hyperparameter sweep, a batch-inference pass over a large corpus, an RL loop launching thousands of short jobs: each one asks the orchestrator to track a flood of concurrent work. Kubeflow Pipelines runs on Argo, which drives everything through Kubernetes objects and etcd, so every task, retry, and artifact is more state in the cluster's control plane. Under heavy fan-out that's exactly where it strains: etcd pressure, API-server throttling, scheduler lag. The failure mode isn't a clean error; it's a platform that gets slow and flaky precisely when you need it most. So teams throttle their own concurrency to keep the cluster healthy, and the sweep you wanted to finish in an hour runs overnight.
You should be tuning your model, not tuning your concurrency down to keep the cluster alive.
Illustrative. The point is where the ceiling sits: on Kubeflow it's the control plane; on Union it's the accelerators you actually have.
Why the control plane is the wall
Kubeflow: Argo materializes each step as Kubernetes objects backed by etcd. That model is fine for modest pipelines and degrades under the high-cardinality fan-out AI produces: the orchestrator, not the GPU, becomes the limiter.
Union (Flyte 2): a purpose-built execution engine, removes the dependency on etcd. Instead, leveraging an engine built in Rust, engineered for 1M task fanout, ~1M concurrent actions, ~1k/s submission, and 300k+ actions per run.
04 · Many clusters, one queue
Your GPUs are in more than one place
Nobody buys accelerators in one tidy block. There is the reserved capacity, the spot pool, the second region, the on-prem rack, the cloud you went to because it had H100s. The question is whether all of that is one pool your teams draw from, or several platforms your team runs.
Kubeflow's answer is per-cluster. A KFP backend is installed into a cluster and its pipeline runs execute there, so a second cluster means a second KFP to deploy, secure, upgrade and watch. For training specifically there is a genuine cross-cluster story, and it is a good one: Kueue's MultiKueue puts a manager cluster in front of worker clusters, and a TrainJob submitted to the manager is mirrored onto a selected worker cluster, where the pods are actually created. For fungible GPU-training capacity that is exactly the right tool. Two things it is not. It dispatches a job, not a pipeline, so your DAG, retries, lineage and data movement do not travel with it. And the training runtime has to be installed and kept consistent on every worker cluster, which is one more thing you reconcile by hand.
Union's answer is that routing belongs to the platform, not to your code. Clusters are registered with the control plane and grouped into pools that share one data plane: the same object store, secret store and container registry. That shared data plane is the whole trick, because it is what lets a workload land on any healthy cluster in the pool and still find its inputs, its code and its secrets. On top of pools sit queues: named scheduling lanes that both route work and govern it. You submit to a queue. You never pick a cluster.
Adding capacity should be a registration, not a second platform.
# each cluster you want pipelines in gets its own KFP # backend to deploy, secure, upgrade and watch kubectl apply -k "github.com/kubeflow/pipelines/manifests\ /kustomize/env/dev?ref=$PIPELINE_VERSION" # training can cross clusters, via Kueue MultiKueue. # the runtime must exist on EVERY worker cluster: kind: ClusterTrainingRuntime # all workers, in sync --- kind: TrainJob # submit to the manager metadata: labels: kueue.x-k8s.io/queue-name: multikueue-queue # Kueue mirrors this onto one selected worker cluster. # pipelines still run where their own backend lives.
# clusters register into a pool sharing one data plane, # so any healthy cluster in it can run any of its work @env.task(queue="gpu-h100") # pin one step to a lane async def train(cfg: Config) -> Score: ... # or choose at launch, with no change to the task flyte.with_runcontext(queue="eu-west").run(main) # or per invocation, decided while the run is going train.override(queue=lane)(cfg=cfg) # queue -> pool -> any healthy, enabled cluster in it
What a queue actually governs
Where the work lands. A queue lives in exactly one pool. By default it spreads across that pool's clusters; you can also pin it to named ones. It can never reach a cluster in another pool, because pools are the isolation boundary. Registering a cluster also creates a queue named after it, so every cluster is addressable by name from the moment it joins, with no queue setup at all.
What happens when a cluster goes bad. Routing is evaluated against current state, not a static list: an unhealthy or disabled cluster receives no new work from any queue, pinned or not, and the pool's remaining healthy clusters absorb it. This is the durability story applied to capacity rather than to a single action.
How the work is rationed. Each lane carries run concurrency (stop a scheduled job overlapping itself), action concurrency (bound the burst rate), depth, priority and fairness. Depth is the interesting one: when a queue is full, new submissions are rejected immediately with RESOURCE_EXHAUSTED rather than piling into an unbounded backlog, which turns a flood into an explicit back-pressure signal your submitter can act on. Priority orders work across lanes contending for the same capacity; it does not preempt what has already started.
Read this fairly
Neither system stretches one synchronous distributed job across clusters, and you should be suspicious of anything that claims to. MultiKueue selects a worker cluster and runs a mirrored copy there; a Union queue routes a workload to one healthy cluster in its pool. Cross-cluster training is a networking, storage and identity problem before it is an orchestration one.
The difference worth arguing is not capability, it is where the routing lives. On Kubeflow it is a second system you assemble around the orchestrator and keep in sync across clusters. On Union it is one parameter on a task, resolved by a control plane that already knows which clusters are healthy.
05 · The sweep
A Bayesian sweep is a loop, not a graph
Hyperparameter search is where the compiled-DAG model shows its seams first, because a real sweep decides its next trial from the results of the last one.
Grid and random search compile fine: the candidate list is known up front, so ParallelFor fans it out and you are done. Bayesian and population-based search do not work that way. The optimizer proposes a candidate, sees the score, and proposes the next one accordingly, which means the shape of round two depends on the outcome of round one. On Kubeflow you either adopt Katib as a second system with its own CRDs and lifecycle, or you run the optimizer outside the pipeline and drive it from the side: compile a round, submit it, poll for results, feed them back, compile again. That outer loop is glue your engineers own and operate. On Flyte 2 the ask, score and tell cycle is ordinary async Python inside a task, so the whole sweep is one run with each trial an isolated, cached, retryable action.
# pipeline topology is fixed at compile time @dsl.pipeline def sweep_round(candidates: list): with dsl.ParallelFor(candidates) as c: train(c) # shape known up front # the adaptive loop can't live inside, so: while not done: cands = optuna_ask() # outside compile_and_submit(sweep_round, cands) scores = poll_results() # you operate this optuna_tell(scores) # glue you own
@env.task async def sweep(n: int) -> Trial: study = optuna.create_study() async with asyncio.Semaphore(16): for _ in range(n): t = study.ask() # ask score = await train(t) # score (cached, retryable) study.tell(t, score) # tell return study.best_trial # adaptive, runtime-shaped: no recompile, no controller
→ See the full developer-experience teardown: one Bayesian sweep, step by step
Why the sweep is the tell
Katib is a capable optimizer and the right answer for plenty of teams. The question is not whether Kubeflow can tune, it is where the tuning loop lives: inside the pipeline as ordinary code, or beside it as a component with its own CRDs, its own release cycle, and its own failure modes to learn. Every adaptive workload that follows, RL and agents included, asks the same question and gets the same answer.
06 · The agent wall
Your roadmap says "run agents."
An agent decides its next step while it runs. A compiled DAG has to know every step before it starts. That is not a gap you close with configuration, it is the difference between the two models.
The sweep above is the mild version of this problem: a loop whose next round depends on the last. An agent is the hard version, because the tool calls, the branching, and the number of turns are all discovered at runtime. On KFP the adaptive controller has to live outside the pipeline, driving it from the side or recompiling every round, and that outside-the-pipeline glue is what your engineers end up owning and debugging. On Flyte 2 the loop is async Python inside a task, and Union ships an agent harness on top of it rather than leaving you to hand-roll the tool-call loop.
What the agent harness gives you
flyte.ai.agents.Agent takes a set of tools and instructions and drives the LLM and tool-call loop for you: send the conversation and tool catalog to the model, execute any tool calls it returns, append the results, and repeat until it replies in plain text or hits max_turns.
What counts as a tool is the part that matters here. Alongside plain Python callables, a tool can be a @env.task durable task or a remote task reference, so a single tool call can be a retried, cached, GPU-scheduled action rather than a function call your process has to babysit. MCP servers (Slack, GitHub, Linear, filesystem) load into the catalog as first-class tools, memory persists across runs, and human-in-the-loop approval can pause the loop before a sensitive tool executes.
Then there is the part nobody puts on the roadmap: an agent that writes code is running code nobody reviewed. The model does not need bad intent to produce a DELETE FROM orders WHERE 1=1, an unbounded loop, or a line that reads your environment variables and posts them somewhere. Running LLM-generated code unsandboxed means trusting the model never to make those mistakes, which is not a control you can show a security reviewer.
Two sandboxes, chosen by what the code needs to do
The workflow sandbox runs the model's orchestration logic on Monty, a Rust-based sandboxed Python interpreter. It starts in microseconds and allows pure control flow only: no imports, no file I/O, no network. Dangerous operations are not forbidden by policy, they are structurally impossible. Heavy work is dispatched out to real container tasks through the Flyte controller, so the generated code stays confined while the tools it calls remain durable, observable and type-checked.
The code sandbox covers the other case, where you genuinely need a full Python environment: third-party packages, file I/O, shell commands. It builds an ephemeral container on demand from declared dependencies, runs once, and discards it.
This is a capability rather than a Kubeflow shortcoming: sandboxing untrusted model output sits outside KFP's scope, so on that stack it is another layer to choose and operate, whether that is a session sandbox such as E2B or Daytona, or your own container isolation.
The last piece is operational, and it is the one teams underestimate. Agent runs are long, expensive, and non-deterministic, so the question is not only whether the loop can be expressed but what happens when it fails at turn forty. Restarting from zero burns the GPU hours again and may not even be successful.
When a long run goes wrong, three verbs instead of one
- Recover (
flyte rerun --recover) launches a new run that reuses the actions that succeeded and re-executes only what failed. A reused action is not run again at all; its recorded output is handed straight to whatever consumes it. Unlike caching, which is content-addressed and opt-in per task, recovery is scoped to the one run you name and needs nothing marked in advance. - Fork (
flyte fork <run> main.py main) replays a prior run against the code and inputs you have now: successful actions are reused, and only what your change actually affects re-executes. The docs call it time-travel debugging for a pipeline. You can change one input without restating the other four, and force a specific action to re-run. - Debug opens a live session on the action itself: one click in the UI, or
debug=Trueto start a browser-based VS Code session inside the task pod, stepping through your code against the same data, images and dependencies the run uses. SSH attach is in beta. This is the answer to a task that behaves differently remotely than it does on your laptop.
Compile-time topology vs runtime execution
This is the same divide behind the durability story: a system that fixes the graph before running cannot recover, branch, or adapt at runtime because it has already committed to a plan. A runtime that executes real Python decides its shape as it goes, which is what agents, RL, and human-in-the-loop approvals require.
07 · What "good" looks like
One runtime that keeps your runs alive
Strip away the seven components, the upgrade treadmill, the idle GPUs, and the compile step, and the alternative is simple to state: one durable, infra-aware runtime, in your own cloud. This isn't "leave Kubeflow"; it's the shape of a platform that doesn't cost you those four things.
The center of it is one word: durability. Not a feature, a property: the runtime answers each of the three ways an AI workload dies.
Durable under failure
Self-heals infra, not just logic
An OOM retries with more memory; a preempted node recovers without paging anyone. Orchestrators recover your code; a runtime recovers your compute.
Dynamic under non-determinism
Runtime-shaped, agent-native
Pure Python that branches, loops, and adapts while it runs: agents and RL are native, not glue bolted onto a compiled DAG.
Native through to serving
Train and serve, one program
Batch and real-time inference in the same codebase and lineage as training: the endpoint inherits the same recovery. No separate serving stack.
Because it's one runtime and one control plane, the consolidation is real: one on-call, one upgrade path. And like Kubeflow, it runs on your own infrastructure: Reusable Containers keep GPUs hot, and RBAC and custom SSO arrive as product rather than per-component assembly, with the identical code you run in your VPC also running on your laptop.
That last part matters at the point a security review starts, which is where consolidation usually stalls. Union is SOC 2 Type II certified for Security, Availability and Processing Integrity, with practices aligned to ISO 27001 and CIS benchmarks, and lists HIPAA among its compliance programs. The report, a BAA, and the underlying control set are published in the Trust Center rather than described in a sales deck. On a composed Kubeflow stack the same review covers every component you assembled, and the evidence is yours to produce.
08 · Getting there
A ramp, not a forklift
The switch people brace for is a big-bang migration. This isn't that. You wrap what you have and move the pipelines that hurt most, first.
01
Bring what you have
Wrap existing tools and legacy jobs as container tasks: one task definition, no rewrite into a new framework to start seeing value.
02
Author in Python
Rebuild the pipelines that hurt most first: the dynamic, high-fanout, GPU-heavy ones a compiled DAG fights you on.
03
Run in your cloud
Local → remote in your own VPC. Add reusable containers, multi-cluster, RBAC and cost visibility as you scale.
| What you leave behind | What you keep or gain |
|---|---|
| Seven components + Istio/Dex/Argo to operate | One runtime, one control plane, one on-call |
| Compiling a DAG before it runs | Pure Python that adapts at runtime |
| Cold pod per task; idle GPUs | Reusable warm containers; hot GPUs |
| A separate KServe serving stack | Batch + real-time inference in the same program |
| Vendor lock fears | OSS Flyte 2 (Apache-2.0, fully GA): identical code runs OSS and on Union |
The open-source floor is the safety net: because Flyte 2 is Apache-2.0 and fully GA, the same code runs in OSS and on Union. You keep the open foundation and add the operated runtime (the scale ceiling, reusable containers, multi-cluster, enterprise plumbing) only where you need it. No rewrite to adopt, no rewrite to leave.
Worth being precise about what that means, because "no rewrite to leave" is only reassuring if you know what you would be leaving behind. The published Flyte 1 / Flyte 2 / Union breakdown sets it out row by row: the authoring model is identical across all three, so your workflow code is genuinely portable, while the differences are runtime, how far it scales, how fast it starts, and how much infrastructure your team owns.
→ Compare Flyte 1, Flyte 2 and Union feature by feature
09 · Like for like
Required Kubeflow & other open-source tools
Union's surface arrives as one product. To reach the same surface on Kubernetes, a team assembles the components below, then owns the integration, the upgrade order, and the on-call for each. This is the honest shape of the trade-off: not that Kubernetes cannot do the work, but that someone has to select, wire, secure, and keep these pieces in step.
Orchestration & execution2 tools
Union: one runtime, one SDK, one control plane.
- Kubeflow Pipelines (KFP v2)compiled pipeline DAGs, typed artifacts, caching
- Argo Workflowsthe workflow engine KFP v2 executes on
Distributed compute5 tools
Union: Spark and Ray orchestrated from the runtime; resources set per task.
- Kubeflow Trainerdistributed PyTorch / TensorFlow / XGBoost training
- MPI OperatorMPI-style multi-node jobs
- Spark OperatorSpark job and cluster lifecycle
- KubeRayRay cluster lifecycle
- Kueue + MultiKueuejob queueing and quota, and dispatching training jobs across clusters
Tuning & search1 tool
Union: sweeps as ordinary runtime Python; Hydra plus Optuna integration.
Serving & inference3 tools
Union: batch and real-time inference native, in the same program as training.
Artifacts, lineage & registry3 tools
Union: typed artifacts, caching and Artifact Hub built in; trackers integrate.
- ML Metadata (MLMD)execution and artifact lineage, built into KFP v2
- Kubeflow Hubmodel versioning and promotion (formerly Model Registry; now covers Model Registry and Model Catalog)
- MLflow or Weights & Biasesexperiment tracking (external on both sides)
Developer experience3 tools
Union: DevBox plus a managed image builder from the SDK.
- Kubeflow Notebookshosted development environments
- BuildKit, Kaniko or cloud buildcontainer image build inside the cluster
- Registry plus CI/CDpublish images and deploy pipeline definitions
Identity, networking & security6 tools
Union: OIDC/SSO, fine-grained RBAC and both agent sandboxes in the product, on a documented zero-trust model: your data, code and logs never reach Union's control plane, which holds orchestration metadata only, and the sole path between the two is an outbound-only connection from your own data plane.
- Istioingress, service mesh, mTLS between components
- DexOIDC identity provider
- OAuth2-proxyauthenticating gateway in front of the UIs
- cert-managerTLS certificate lifecycle
- Vault or a cloud KMSsecret storage and rotation
- E2B or Daytona, or your own gVisor isolationsandboxing untrusted, model-generated code
Observability & cost3 tools
Union: per-project and per-domain cost and utilization reporting native.
- Prometheusmetrics collection
- Grafanadashboards
- OpenCost or KubecostGPU and workload cost attribution
Read this fairly. Few teams need every layer: scope it to the workloads you actually run, and a pipelines-only deployment needs a fraction of this. Several entries (Prometheus, Grafana, cert-manager, Istio, a registry, CI/CD) are general Kubernetes platform tooling many teams already operate, so they are not Kubeflow-specific overhead. Knative applies only if KServe runs in Serverless mode, and where alternatives are listed you pick one rather than all. Union is not zero on every row either: experiment tracking is an external integration on both sides. The point is not the count, it is who owns the integration and the upgrade path.
10 · Common questions
What teams ask before they move
How many components does Kubeflow actually require?
Kubeflow Pipelines (KFP) can be deployed on its own, and a team that only needs a DAG runner should do exactly that. The weight described on this page is the full distribution: six governed subprojects (Pipelines, Katib, Trainer, Notebooks, Spark Operator, Hub) plus KServe for serving, which Kubeflow lists in its Ecosystem rather than governing as a subproject, on top of Istio, cert-manager, Dex, OAuth2-proxy and Argo Workflows, with Knative as well where KServe runs in its serverless mode. Most teams adopt the distribution because they want the platform rather than just the orchestrator, and that is what carries the operating cost.
Does Kubeflow have GPU utilization features?
Not in the orchestrator itself. Warm container reuse, dynamic batching and GPU packing come from the serving layer you add on top, typically KServe, Ray Serve, vLLM or Triton. The workload is achievable either way; the difference is whether GPU efficiency is a platform feature you switch on or another service to select, integrate, secure and upgrade. Union ships reusable containers and DynamicBatcher as part of the runtime.
Can Kubeflow Pipelines run agents or reinforcement learning?
KFP compiles a static DAG before execution, so the shape of the work has to be known up front. Adaptive patterns that decide their next step at runtime need an external controller driving the pipeline, or a recompile each round. On Flyte 2 the same loop is ordinary async Python inside a task, so the ask, score and tell cycle stays in one program with each trial cached and retryable.
Can Kubeflow run across multiple clusters?
Partly, and it depends what you mean. A KFP backend is installed into a cluster and its pipeline runs execute there, so pipelines do not span clusters on their own. Training does: Kueue's MultiKueue puts a manager cluster in front of worker clusters and mirrors a TrainJob onto a selected worker, where the pods run. That is a good fit for fungible GPU-training capacity, with two caveats: it dispatches a job rather than a pipeline, so the DAG, retries and lineage do not travel with it, and the training runtime must be installed and kept consistent on every worker cluster. Union routes at the run level instead: clusters are grouped into pools sharing one data plane, and you submit to a queue that lands the work on any healthy cluster in its pool.
Is Kubeflow a mature project?
Yes. Kubeflow became a CNCF Graduated project in August 2026, having completed a third-party security audit and established formal steering governance. It is genuinely open, has a large community, and carries no vendor in the loop. The argument on this page is not that Kubeflow is immature or incapable; it is about what the distribution costs to operate, and whether that is the trade your team wants to keep making.
Do we have to rewrite our pipelines to move?
Flyte 2 is open source under Apache-2.0 and fully GA, and the identical code runs on OSS Flyte and on Union. Existing containers and legacy jobs can run as container tasks with one task definition and no rewrite into a new framework. The usual path is to rebuild the pipelines that hurt most first, the dynamic, high-fanout, GPU-heavy ones, and leave the rest where they are.