The first in a series on the engine under Flyte 2.0: what we needed it to survive, the requirements we set for it, and the architecture we ended up with.

"Everything fails, all the time."
— Werner Vogels

New to Flyte 2? Start with the launch post; this series is the engineering behind it.

Over the last twelve months, one Union control-plane instance ran more than three billion actions, at under five percent of its capacity. This post is about what we had to build for that to be true, and why most of it comes down to a single decision about what durability means.

What durable means

Most workflow systems are DAG orchestrators. You hand them a graph and they walk it, and anything you wanted beyond that, such as a retry that changes something, a recovery that reuses finished work, or a way to look inside a step, ends up bolted on around the graph. The ones that describe themselves as durable usually mean durable against code errors, where a retry replays the same code and assumes the machines underneath are still the same consistent set they were a minute ago.

We couldn't build on that assumption, because it isn't true of the infrastructure our users run on. Hardware dies, requests get sized wrong, model APIs flake, and workloads outgrow the machine they started on and need more memory, a different GPU, or more time. So we built Union, the engine that runs Flyte 2.0, around the idea that infrastructure changes under a run. You write ordinary Python, and when a machine dies the run recovers and continues from where it was. When a step turns out to need a different machine, the retry changes the infrastructure instead of replaying the same code on the same box.

So when we say durable, we mean recovering from both kinds of failure, infrastructure and code, without redoing the work that already finished. We care about that second clause as much as the first. Compute is expensive, and once a run gets large enough, recovery stops being a cost optimization, because a run that can't resume from where it died is a run that never finishes.

The verbs compose: cache, fork, recover, rerun, retry. Here's what each one looks like from inside a task.

Caching means an expensive computation runs once, and any later run with identical code and inputs reuses the recorded result by name, across runs and across machines:

@env.task(cache="auto")
async def embed_corpus(docs: flyte.Dir) -> Embeddings: ...
# runs once; identical code + inputs reuse the recorded
# result by name, across runs and across machines

Retries and timeouts let you back off on flaky dependencies, fail fast when a scarce GPU pool is full, and put a bound on the whole attempt:

@env.task(
    retries=flyte.RetryStrategy(
        count=5,
        backoff=flyte.Backoff(
            base=timedelta(seconds=30),
            factor=2.0,
        ),
        timeout=flyte.Timeout(
            max_queued_time=timedelta(minutes=15),
        ),
        deadline=timedelta(hours=2),
    ),
)
async def needs_scarce_gpu() -> Result: ...
# fail fast when the pool is full; never exceed two hours end to end

An out-of-memory kill is an infrastructure error, but you can catch it in code like any other. The retry is the same step in a bigger container, and the replay log doesn't change:

@env.task
async def summarize(source: str) -> Summary:
    try:
        return await scrape(source)
    except flyte.errors.OOMError:
        # same step, bigger container; the replay log is untouched
        return await scrape.override(
            resources=flyte.Resources(memory="64Gi"))(source)

GPU faults we split by who caused them. If the hardware raised an Xid, it becomes a system error attributed to the exact action and gets retried on the platform's budget rather than yours. If your own kernel caused it, it arrives as a typed `GPUFaultError` carrying the Xid, the GPU, and the node that saw it, so you can decide what to do:

try:
    result = await train(shard)
except flyte.errors.GPUFaultError as e:
    # severity "USER" means your kernel faulted; hardware faults
    # were already retried on the platform's budget before reaching you.
    if e.severity == "USER":
        raise
    result = await train(shard)

Scale comes out of the same design. The more a team can do, the more they want to run at once: one team validating an autonomous-driving stack replays millions of scenarios per release, and another training a reinforcement-learning policy runs millions of rollouts between updates. In Flyte 2 you ask for that the way you'd write any Python, with a loop and a `gather`, and the runtime spreads the work across as many machines as it needs:

@env.task
async def eval_policy(cfg: Config) -> Score: ...

@env.task
async def sweep(cfgs: list[Config]) -> list[Score]:
    # fans out across thousands of machines; a million rollouts, no re-architecture
    return await asyncio.gather(*[eval_policy(c) for c in cfgs])

The dispatch is fast enough that this works for agents as well as for batch jobs. An action goes from zero to scheduled, meaning the decision is made and a lease is on its way to a worker that starts the container, in under a millisecond, which is what lets an agent's next step feel immediate.

Around every run you also get the things production needs that have nothing to do with your code. Cached tasks mean identical work is never paid for twice, across runs and across machines. Forking a previous run lets you replay yesterday's run with today's code or a changed input, reusing everything the change doesn't touch. You can watch a run in progress per action, with its inputs, outputs, metrics, and logs as they arrive. One control plane can find machines in other clouds or regions and pool them together. And queues let you cap what runs in parallel, whether by run, by action, or by resource, because your downstream systems have limits even when the platform doesn't.

Nine requirements

We started with the language. If you know Python, you can build on this: functions, loops, `async` and `await`, and one decorator. There's no DSL and no graph to compile, and fan-out is a loop and a `gather`. Part of that was ergonomics, but we had a second reason. Language models are good at writing natural code and much worse at learning a framework's inverted patterns, so we didn't want anything turned inside out for a person or a model to work around. We also assumed Python wouldn't be the last language, which meant the engine had to be indifferent to what wrote the work. A Rust or TypeScript task can call a Python one and the run doesn't care.

The second requirement was that each function could take its own shape at runtime, meaning its image, its resources, its accelerator, and whether it shares a warm container, and that all of it would be configured on its Environment right next to the code:

env = flyte.TaskEnvironment(
    name="trainer",
    resources=flyte.Resources(cpu=8, memory="64Gi", gpu="H100:8"),
    image=flyte.Image.from_debian_base().with_pip_packages("torch"),
)

@env.task
async def train(shard: int) -> Checkpoint: ...

That declaration travels with every piece of work. It's what later lets a retry change its infrastructure, and it's how the scheduler knows what an action needs before any pod exists.

Third, the interception had to be durable. The runtime intercepts the code's flow and runs each function on its own container, and where a fresh container would cost too much it lands on a warm pool instead. Where distributed execution would cost more than it buys, traced functions run in-process as actions. Each execution of a task is an action, the tree of actions triggered from one primary action is a run, and every action gets a deterministic name. Those names are how recovery, caching, and observability agree on what happened.

Fourth, the engine had to make good decisions on shared, finite compute. Almost every customer is a team, or many teams, sharing a fixed pool of GPUs, which means every scheduling decision is also a decision about who waits. The engine had to see what each action needs and what each cluster has, hold gangs together instead of letting them starve behind small work, and keep quotas per team. The schedulers that already did this made their decisions in seconds, and at hundreds of thousands of pods those seconds stretch and the failures go opaque. Since we had users running real-time agents on this, we couldn't accept slow and correct.

Fifth, it had to be multi-cloud and multi-cluster with zero trust: one control plane scheduling over many Kubernetes clusters, across clouds and on-prem, without any data plane ever being opened up to the control plane. Workloads move between clusters and are routed on resources, and heavy workloads such as distributed training, Ray, Spark, and clustered inference had to be carried as naturally as a single container.

Sixth, scheduling had to be resource-aware. The engine had to understand resource configuration, the four dimensions plus accelerators, whether the request was a plain container or a Ray cluster of one head and n workers, and admit work only where it fits.

Seventh, caches had to outlive machines. A follow-on step should land in the warm pool or the Ray cluster its predecessor created, a task that reads a large dataset should run where the data already is, and a cached output should be findable by name from any machine. For all of that to hold, the scheduler itself has to understand affinity rather than treating it as a hint it can drop under pressure.

Eighth, a scheduling decision had to take under a millisecond. We made the scheduler event-driven rather than polling, so it runs the instant an action is enqueued, a worker connects, or capacity frees. That budget shaped most of the design that follows, because you can't wait on a database inside a decision that fast.

And ninth, no runaway compute. When you abort a run, the abort has to reach everything the run started: the training job, the remote Databricks or SageMaker run, every child of a parent that failed. It also had to cost zero boilerplate, because the people on this platform are researchers and scientists from every field, and cleanup they have to remember to write is cleanup that mostly won't get written.

Three more, for ourselves

Operating a high-scale, multi-cloud scheduling service is its own engineering problem, one that stresses networking, auth, and performance at the same time. It's also a business, so the cost of running the platform has to start small when a region has few customers and grow with usage rather than ahead of it. Three more requirements came out of that.

The first was scaling dynamically, toward billions of actions. We set the goal as a number that keeps moving, billions of actions and beyond, with no re-architecture on the way up, and that shaped every choice in the backend. Capacity in a region starts small and grows as customers arrive and as they scale, and every service gets there by adding machines.

The second we called sleep. We run this as a service, for many customers at once, on infrastructure we operate, so the engine had to be multi-tenant from the first line, with one tenant's million-action backfill never becoming another tenant's problem. It also had to scale without a person in the loop: no resharding weekends, no manual failover, no operator editing rows in a database at three in the morning. Whenever we had to choose between a clever design and one we could leave alone, we picked the one we could leave alone.

The third was letting users troubleshoot in private. When a run fails at step 40 of 60, the person who owns it should be able to see exactly what happened in their own environment, without filing a ticket and without their data leaving their account: the inputs, the outputs, the logs, the error and its cause, and why an action is still queued. Support that starts with "send us your logs" doesn't scale, and neither does the version where we're in the loop for every failure. More and more, an agent does the first pass of that debugging automatically, over the same APIs.

The architecture

The design splits along one line. Taking in a flood of writes from hundreds of thousands, or millions, of active runs is one job, and making a single, consistent scheduling decision from a complete picture of queues, workers, and capacity is another. Those two pull in opposite directions, and separating them is what lets the engine scale both at once, each in its own dimension.

The action store is the write path. It's highly sharded and distributed, and it's the only thing a running task ever talks to. Every child task, every traced function, and every wait for a human is recorded there durably before the call is acknowledged. Each shard handles its own writes with no coordination between shards and no scheduling logic in the way, which is why the write path scales out by adding shards. It persists into the run store, the durable history of every run, whose outputs live in your object store. A cache service resolves cache hits by name for the workers, and the run store feeds the analytics store behind the dashboards. The UI and APIs sit on top of all of it, watching live runs over the same streams the runtime uses, and recovery and caching are both lookups by name against that history.

The scheduling engine is the decision path. We sharded it so that everything one scheduling decision needs, the clusters it may place onto, the queues, the concurrency limits, and the capacity, lives with a single owner. Each shard belongs to exactly one leader-elected process that holds the shard's complete state in memory and decides without a lock and without a round trip. Durability comes from leasing semantics. A lease is a time-bound grant of one unit of work to one worker, written down before the worker hears about it, expiring unless the holder heartbeats, and fenced by a generation number. A crashed worker, a partitioned cluster, and a deleted node pool all turn into the same event, a lease that stopped heartbeating and got reassigned. Cleanup rides on the same mechanism: finalize leases are sticky and routed back to the cluster that ran the work, so an abort tears down everything a run started, wherever it started it. The docs describe leases from the user's side.

The per-run controller lives in the SDK and runs inside your task's process, as a Rust core with Python bindings. It names each child action deterministically, records it, and watches for results the way a Kubernetes controller does, with streams and level-triggered reconciliation rather than polling. The names it computes are the primitive everything else keys on.

The cluster worker runs inside each data-plane cluster and scales horizontally with it. It dials out to the engine, executes leases through plugins, and heartbeats everything it holds. Above all of this sits the observability layer, which gives you both the real-time view of a live run and the analytical view across runs.

When a node dies

The pieces are easier to see composed than described, so take the case the Vogels quote is about. A large fan-out is in flight, and the node running one of its actions disappears.

Before that action ever started, the per-run controller in the parent's process had already named it deterministically and recorded it in the action store, and the parent's call was acknowledged only once that write was durable. The scheduling engine's shard for the run then wrote a lease granting the action to one worker, fenced with a generation number, and the cluster worker began heartbeating it.

When the node dies, the heartbeats stop. The engine doesn't need to know whether it was a crash, a partition, or a node pool someone deleted, because all three look the same from where it sits: a lease that stopped heartbeating. It reassigns the action under a new generation, so anything the old holder tries to report late is fenced off. If the failure was a hardware fault such as a GPU Xid, the retry counts against the platform's budget rather than the user's.

The parent doesn't poll for any of this. Its controller watches the action's state over the same streams the UI uses and reconciles when it changes. Because the action's name is deterministic, the new attempt is the same action, and every sibling that already finished is untouched: their outputs are in the run store, findable by name, so nothing that completed is recomputed. If the retry needs a different machine, that override travels with the action's declaration and the scheduler places it accordingly.

If instead it's the parent's own process that dies, recovery works in the other direction. A new parent replays the code, and each child call resolves against the recorded action by name instead of running again. That's what the replay log is. And if the control plane itself is unreachable, the controller retries with exponential backoff for roughly twenty minutes before giving up, which is long enough to ride out most outages without a person noticing.

Where it stands

• 3B+ actions per year on one control-plane instance today, at under 5% of its capacity
• 1M actions in a single run; 100M+ per org
• ~52k actions per second dispatched by one scheduling-engine shard (benchmark)
• 0.6 µs scheduler cost per pending action per scheduling pass

The benchmark figures are single-shard, in-memory measurements on a laptop-class machine, and the appendix lists the exact configurations. The production figure is measured rather than projected.

What customers say

"Flyte 2 it's just so much better. We can scale to 200,000–300,000 pods with the escalation logic baked right in, and the out-of-memory and scheduling headaches I used to fight are simply gone."
— Jay Ganbat, Principal Bioinformatics Engineer, Prima Mente

"Our inference runs exceed the scale limits of a standard EKS cluster. With Union, we can have a single run span multiple clusters while having that single run spawn thousands of GPUs and call hundreds of thousands of actions, all of which are cached durably."
— Hariharan Ananthakrishnan, Principal Engineer, Artera AI

"Nate and I built an identical inference pipeline at our last company, but using Argo Workflows. It was a lot more painful — the Workflow-of-Workflows pattern, which sucked."
— Jeff Albrecht, Head of Engineering, LGND

"If we weren't using Union, we would be writing our own Kubernetes Karpenter stuff that probably doesn't work as well."
— Jeff Albrecht, Head of Engineering, LGND

"Initially it was something like four minutes, because we were pulling every layer. Now if I look at the logs, it's always under one second. We never have any issue with cold starts."
— Arka Purkayastha, Research Engineer, Third Dimension AI

"v2 definitely easier to scale than Ray, since you have a lot more granular control over streaming the parallelism. The parallel async model is nice to work with."
— Patrick Surry, Chief Data Scientist, Hopper

Next up

Flyte 2 gave up the compiled graph so a run could be plain Python, which means the only complete description of a run lives inside a process that is allowed to die. The next post is about how we made that durable anyway: a write-ahead log built while the code runs, where every child task, traced function, and wait for a human becomes an entry under a name a fresh process can compute again. Replay, recover, and fork are three reads of that log, so a lost node, a bug fix, or a failing model API costs one step rather than the whole run. It walks through the per-run controller that builds the log, how each action's name is computed and what is deliberately left out of it, and what was hard to get right.

If you'd rather see the whole path from the user's side first, the life of a run page walks it end to end. And if you'd rather run it than read about it: `pip install flyte` and `flyte start devbox`.

Appendix: numbers and configurations

Every figure in this post, with the setup behind it. The first one is production. The rest are single-shard benchmarks, in memory unless a row says otherwise, so read them as the ceiling for one shard rather than as fleet numbers. Scaling out is a matter of adding shards, which is the whole point of the split described above.

Production

Scheduling engine, one shard

Action store, one shard

Control-plane outage tolerance

The per-run controller retries a system error up to 100 times with exponential backoff capped at 10 s. That works out to roughly 20–25 minutes of continuous control-plane unavailability before a run fails.