Anatomy of a Durable Run

A tour of the four durability mechanisms in the open-source AI runtime — intra-task checkpoints, crash recovery, cross-run recovery, and caching — with the code and the design decisions behind each.

The term "durable" does a lot of work in AI and Agentic applications, taking on different meanings in different contexts. In reality, durability isn't just one mechanism. It's actually it's four, each scoped to a different blast radius.

A spot node dying mid-epoch is a different failure than the workflow driver crashing, which is different from a run failing on the last task due to API rate limiting, which is different from re-running work someone else already computed. Conflating them is how you end up with a system that retries beautifully and still loses nine hours of GPU time.

In this post, we'll step through the four layers of durability.

Two architectural facts about Flyte 2 underlie all four layers:

• It externalizes run state here a failed process can easily re-hydrate state at a granular level
• It exposes infrastructure-as-context, providing methods to observe and modify compute at runtime.

A run isn't one object owned by one process; it's many small per-action records, reconciled by a worker pool, with outputs offloaded to durable storage. Every mechanism below is a different way of leveraging that.

Layer 1: surviving failure inside a task attempt

The smallest blast radius is a single attempt of a single task: the pod gets preempted, the GPU flakes, the process dies at epoch 47 of 50. Retries alone would restart at epoch 0. Checkpoints make the retry resume instead.

Any task that declares `retries` gets a checkpoint object scoped to the action. Each attempt can read what the previous attempt saved:

# The pattern, in pseudocode

# This is None on the first attempt, but in subsequent
# attempts this path will point to state saved by checkpoint.save
path = await checkpoint.load()
start = int(path.read_bytes()) if path else 0
for i in range(start, n_steps):
    do_one_unit_of_work(i)
    await checkpoint.save(str(i + 1).encode())

`save()` accepts raw bytes, a file path, or a directory (where directories are compressed into gzip tarballs); repeated saves overwrite the same object-store prefix, one per action attempt, so storage overhead stays minimal. There are `save_sync()`/`load_sync()` variants for synchronous contexts and framework callbacks — PyTorch Lightning, Hugging Face Trainer, and friends.

In real training code the checkpoint carries state relating to the model, optimizer, and progress tracking. This is from a runnable example in the SDK repo (note the `RuntimeSystemError`, which is meant to simulate things going wrong during training):

@env.task(retries=3)
async def train(num_epochs: int = 20, ...) -> list[EpochMetrics]:
    ckpt = await load_checkpoint(ckpt_path, config)
    if ckpt is not None:
        meta, model_state, opt_state = ckpt
        model.load_state_dict(model_state)
        optimizer.load_state_dict(opt_state)
        print(f"Resuming from epoch {meta.epoch}")

    for epoch in range(meta.epoch + 1, num_epochs + 1):
        ...  # train one epoch
        await save_checkpoint(ckpt_path, meta, model, optimizer)

        # Simulated failure at epoch 10 on the first attempt only.
        # On retry the checkpoint from epoch 10 exists, so training resumes at 11.
        if epoch == 10 and get_attempt_number() == 0:
            raise flyte.errors.RuntimeSystemError("simulated", "failure after epoch 10")

For the in-process steps of an agent loop there's a lighter-weight tool: `@flyte.trace`. A traced function's result is recorded as it completes, so when a workflow is re-executed after a failure, completed traced calls replay from the record instead of re-running. An LLM call you paid for once doesn't get paid for twice:

@flyte.trace
async def plan(history: list[str]) -> str:
    ... # your LLM call 

@env.task
async def agent(goal: str) -> list[str]:
    history = [goal]
    # a crash leads to immediate retrieval of
    # saved plan invocations given the same inputs
    while (step := await plan(history)) != "done":
        history.append(await act(step))
    return history

An agent workflow in the Flyte UI: the `while` loop is ordinary Python, and every `plan`/`act` step is a durable, inspectable action.

Layer 2: surviving failures inside a run

The next blast radius is the run itself. Two mechanisms live here, and they cover the two directions failure comes from.

The engine can crash. Flyte 2 maintains a replay log of task start and completion events. If the process driving a workflow dies, it restarts, replays the log, and reconnects to completed and in-flight actions. This is silent, automatic, retried up to roughly 20 times on system errors, and requires nothing from your code. It's also why `while` loops and `asyncio` are safe as workflow control flow: the loop's progress is reconstructible, so the driver process isn't a single point of failure.

The infrastructure can fail. This is where infrastructure-as-context get real. A task killed by the out-of-memory killer doesn't surface as a generic crash; it surfaces as `flyte.errors.OOMError`, in-process, where your code can catch it and re-run that step with a bigger resource envelope:

@env.task
async def drive(tiers: list[int], required_mb: int) -> int:
    for mb in tiers:
        try:
            return await worker.override(
                resources=flyte.Resources(memory=f"{mb}Mi")
            )(mb, required_mb)
        except flyte.errors.OOMError:
            continue          # bump memory, re-run the SAME step
    raise RuntimeError("OOM at max tier")

From the task's perspective it called a function and got a result. From the runtime's perspective, a pod exceeded its memory limit, was killed by Kubernetes, and was re-dispatched with a larger request.

Infrastructure failures become part of the control flow 🤯

In Flyte 1, retries were declared statically on the graph before the run started — there was no way to catch a live `OOMError` from a specific node and relaunch that exact node with more memory. This pattern was one of three that scored "infeasible" in all six v1 trials of our agent-authoring benchmark, and ordinary Python in v2.

The Flyte UI during a failure: the failed action retries with adjusted resources while completed work stays completed.

Layer 3: surviving failure across runs with recovery

Sometimes a run terminates anyway. It could be a genuine bug in one task, a fault that exhausted its retries, or a timeout. The failure is rarely the whole run. Yet, the traditional move is to start a brand-new run with no memory of the old one, re-executing hours of GPU time and expensive data loads that already succeeded.

`recover` fixes this: it launches a new run that references a prior run, reuses every action that succeeded there, and executes only what failed or changed.

import flyte

# Reuse r1's successes to recover
flyte.rerun("r1", main_task, recover=True)

# Same thing from the CLI:
flyte rerun --recover r1 main.py main_task

The interesting part is how the new run knows what to reuse, because Flyte 2 graphs are dynamic — built at runtime by `async`/`await`, with shapes that can depend on runtime values. There's no static DAG to diff. Instead, every action's name is a deterministic hash:

def new_sub_action_from(self, task_call_seq, task_hash, input_hash, group):
    """Make a deterministic name"""
    components = f"{self.name}-{input_hash}-{task_hash}-{task_call_seq}"
    ...

The name folds together the parent action, the task's code hash, the input hash, and the call position. This results in the following:

• It's stable across runs. Same code, same position, same inputs → same name in any run. That shared name is the matching key.
• It's sensitive to change. Edit a task → its code hash changes → its name changes → no match → it runs fresh. Its output changes, so every downstream action's input hash shifts, and the affected subtree re-runs with it. Unchanged siblings keep their names and are reused.

The deterministic name is the diff. At the moment each action is enqueued, the platform looks up its name in the reference run and decides:

Some properties worth calling out:

• It's version-aware without versioning ceremony. Temporal recovers by replaying event history against the same workflow code, which forces `GetVersion`/`patched` markers on every change. Airflow's "clear status and rerun" is version-unaware and destroys the original run's history. Metaflow's `resume` is the closest analog. `recover` needs no version markers — changing a task changes its name — and it produces a new run with full history, with the original untouched and a lineage link back.
• It composes with approvals. A human-in-the-loop signal that was already answered in the reference run is reused and the approver isn't re-prompted.
• It fails openly. If the recovery store is unreachable, actions simply run fresh. Recovery is an optimization, never a correctness dependency.
• `flyte.rerun("r1")` covers the case where you don't have the code locally: it fetches the reference run's original entrypoint and recovers from it, optionally substituting a fixed task via `task_template=` to validate a fix against the exact inputs that triggered the failure.

Layer 4: caching — durability across all runs

Recovery reuses one specific run. Caching generalizes that: deterministic work with unchanged code and inputs never recomputes, in any run, by anyone in the project.

@env.task(cache="auto")
async def preprocess(data: File) -> File:
    ...  # cache version derived from the function body; edit → auto-invalidate

@env.task(cache=flyte.Cache(behavior="override", version_override="v1.2"))
async def featurize(data: File) -> File:
    ...  # version pinned by hand; survives refactors that don't change logic

The cache key combines the inputs, the task's qualified name, the interface hash, and the cache version. It also exposes useful controls on top:

• `ignored_inputs=("debug_flag",)` keeps incidental parameters out of the key
• `salt` partitions cache namespaces for experiments
• `serialize=True` ensures identical concurrent calls run once, and mapped tasks cache per-element, so one changed item in a fan-out doesn't invalidate its siblings. `flyte.with_runcontext(overwrite_cache=True)` forces recomputation for a run.

Caching and `recover` sound similar and are deliberately different tools:

Rule of thumb: caching is for work that's generally reusable; `recover` is for resurrecting this specific failed run without touching a shared namespace.

Does the durable architecture cost anything?

We run a suite of benchmarks to test out the outcome Flyte 2's durable architecture compared to Flyte 1.

In the performance benchmark, on identical 8 GiB pods, the externalized-state engine runs the common patterns 4.3–6.5x faster than Flyte 1's in-process design, and its memory stays flat around 0.3 GiB where v1 grows with workflow size until it's OOM-killed near a 6,000-task fan-out.

In a separate study, a coding agent reached working pipelines in 1.78x fewer tokens on v2 — and the three patterns v1 couldn't express at all are the ones this post is about: catch a live OOM, race and cancel, checkpoint a loop.

The benchmarks and raw data are public and `uv run`-able against your own cluster.

All the primitive building blocks in one

Remember, a run that loses its work when infrastructure fails isn't a run, it's a bill you could have avoided. The four layers above are what it takes to make that the AI runtime's problem, not yours.

Try it

pip install flyte
flyte start devbox

• Flyte 2 GA announcement
• Intra-task checkpoints guide
• Caching guide
• Benchmarks + reproduction scripts