Rerun, Recover, Fork: Three Verbs for a Run That Already Failed

Here's a scenario all too many AI builders encounter: a nine-hour pipeline fails at hour eight, and the first seven hours of work are sitting in object storage, correct and paid for. What you do next is a question about tooling, and for most of the history of workflow orchestration the answer has been some combination of "run it again and go to lunch" and "comment out the first four steps."

Flyte 2 gives you three verbs, and they differ on two things: whose code runs, and which actions get re-executed.

All three commands only work on a remote cluster. There's no local equivalent, because they depend on the platform holding a record of what the source run did.

What a recovery reuses

Take a run whose `summarize` task failed, leaving the downstream `report` task unreached:

A reused action is not executed again at all. It isn't a fast path or a warm cache hit; the recorded output is handed straight to whatever consumes it. Reused actions land in a `RECOVERED` phase, which is terminal and success-equivalent, so `flyte get action ` tells you precisely what the recovery skipped rather than leaving you to infer it from timings.

The parent task always re-executes. It's the code that re-drives the graph, so it has to run in order to make the calls; its children are then matched one at a time as it makes them.

flyte rerun <run-name> --recover --follow

How this differs from caching

Caching and recovery both skip work that has already been done, and they get conflated constantly. They're keyed on different things and they aren't alternatives.

Caching is content-addressed. A task's cache key comes from its inputs, name, interface, and cache version, so a hit can be served to any run at any time within the lookup scope. You opt in per task, in code, and `flyte.TaskEnvironment` defaults to caching disabled.

Recovery is scoped to a single run. It reuses what one named run produced, at the position in the graph where that run produced it. You opt in per run, at launch, and no task has to be marked as anything beforehand.

Three things follow from that, and the middle one is the one people get wrong.

Changing inputs on the way through

The source task's inputs become options on `flyte rerun`, read off the run itself rather than from your local code. Each one is optional, and an input you leave out keeps the prior run's value, so changing one input of a five-input task is one flag:

flyte rerun <run-name> --recover --threshold 0.9

There's a sharp edge here that the docs flag with a warning, and it deserves the warning. An action reused from the source run keeps the output it produced under the original inputs. Recovery does not recompute it against the new ones. If you lower a threshold and the task that consumes it already succeeded, you get the old answer, silently and quickly. Name the actions that must re-execute:

flyte rerun <run-name> --recover --threshold 0.9 --force-rerun-action a3

Action names are deterministic hashes rather than positions, so you list them with `flyte get action <span>&lt;run-name></span>` and copy the one you want. A listed parent re-enqueues its children, so to force a whole subtree you list those too. Unknown names are ignored rather than rejected, which is convenient and also means a typo fails silently.

The rest of the options:

`--action-name` can't be combined with `--recover` for a structural reason: recovery matches succeeded actions by name, and a run rooted at a single action has a different action tree, so there's nothing coherent to match against.

The Python API takes the same arguments:

import flyte

flyte.init_from_config()

flyte.rerun("<run-name>", recover=True)
flyte.rerun("<run-name>", recover=True, force_rerun_actions=["a3", "a7"])
flyte.rerun("<run-name>", recover=True, threshold=0.9)
flyte.rerun("<run-name>", action_name="a3")

flyte.with_runcontext(name="retry-1", env_vars={"LOG_LEVEL": "20"}).rerun("<run-name>", recover=True)

Task inputs share the keyword namespace with those arguments, so a task input named `run_name`, `action_name`, `recover`, `force_rerun_actions`, or `allow_missing_source_outputs` can't be passed this way.

Recovery replays code as-is, on purpose

`flyte rerun` fetches the prior run's task spec and inputs from the platform. No local code is involved, and nothing you edit locally is picked up. This is a deliberate refusal, not a missing feature: recovery exists to survive intermittent infrastructure failures, and a mechanism whose behavior depends on the state of somebody's working tree is not a durability mechanism.

Which leaves the other half of the problem. Most failures at hour eight aren't infrastructure. They're a bug in the code, and you want to fix the bug without paying for the first seven hours again.

Fork: your code, their successes

flyte fork <run-name> main.py main

A fresh code bundle is built from your working tree, so nothing of the source run's code is replayed except where it's unchanged. Actions whose code changed re-execute; everything that succeeded and is unchanged is reused. Go back to a run, change a task, and re-run only what that change affects.

`flyte fork` ships in `flyteplugins-union` 0.8.1 and later, alongside `flyte` 2.6.5. `flyte run`'s options (image, copy style, service account, env, labels, queue) come before the run name:

flyte fork --name fix-1 --env LOG_LEVEL=debug <run-name> main.py main
flyte fork --force-rerun-action a3 <run-name> main.py main

In Python the substitute code is optional, which turns out to be useful:

from flyteplugins.union import fork, with_forkcontext

fork("<run-name>", task_template=main)                        # new code
fork("<run-name>", task_template=main, threshold=0.9)         # new code and inputs
fork("<run-name>", threshold=0.9)                             # source code, new inputs
fork("<run-name>", task_template=main, force_rerun_actions=["a3"])

with_forkcontext(name="fix-1", env_vars={"LOG_LEVEL": "debug"}).fork("<run-name>", task_template=main)

That third line is the way to recover a run with different inputs and no code change at all.

How an action gets its name

Everything about what a fork re-runs falls out of one thing: action names are computed, not assigned.

A fork reuses an action when its action name matches one that succeeded in the source run. Two properties come out of that, and neither is an accident.

Deployment details are excluded on purpose. Changing a task's resources or image doesn't rename its action, so the rest of the run stays reusable. Raise the memory on a task that hit `flyte.errors.OOMError`, fork, and you get what you'd hope for: the successful upstream is reused, and the one failed task re-executes with its new limit. The alternative design, where the image tag participates in identity, would mean that rebuilding a container invalidates an entire pipeline, which is how most systems behave and why most people stop trusting them.

Matching follows the data. A child's name folds in its inputs hash, so an upstream task that re-executes but produces identical outputs leaves its downstream actions matchable, and they're still reused. You fix a logging statement in step two, step two re-runs, step three sees the same bytes it saw before and doesn't.

Here's what a given edit actually costs:

Those last two rows are the ones to write on your hand. A task's identity hashes only the decorated function's own source. Edit a module-level constant it reads, or a helper function it calls, and the action name doesn't change, so a task that already succeeded gets reused and your change has no effect on it. Nothing warns you. You will sit there reading logs from code that didn't run.

The fix is `--force-rerun-action` on the affected actions, or moving the changed logic into the task body where identity can see it. The same rule governs `flyte.trace` functions: a traced function hashes its own body, so editing it makes it re-execute instead of replaying its recorded result, but a helper it calls still doesn't count.

One more case worth knowing: runs launched from a notebook or with a pickled code bundle version on the bundle itself. Any code change renames every action, and nothing is reused. Fork against a file-based run if you want the fine-grained behavior.

Which verb, when

`flyte rerun` when you want a clean run with the same inputs and no local code involved. Cache hits still apply.

`flyte rerun --recover` when the failure was infrastructure and the code is fine. No preparation required, no task needs to have been marked cacheable, and `RECOVERED` in `flyte get action` tells you what you saved.

`flyte fork` when the failure was a bug. Fix the task, fork the run, and pay only for what your change touched.

`--force-rerun-action` whenever you changed something identity can't see: an input that a succeeded action already consumed, a helper function, a module-level constant.

These three are separable at all because Flyte 2 computes action identifiers deterministically from task and invocation context, and keeps them outside the process running the work. A controller uses that to resurrect a run after a crash; `fork` uses it to reach into a finished run and pick up its successes. Same machinery, two problems.

Try it

pip install flyte flyteplugins-union

flyte get action <run-name>                        # see what succeeded, and its action names
flyte rerun <run-name> --recover --follow          # infra failure: same code, reuse successes
flyte fork <run-name> main.py main                 # bug: your code, reuse successes

Full details in Recover a failed run and Fork a run.