From ten thousand simulations. To one real robot.

Union is built on Flyte, the open-source AI runtime we create and maintain under the Linux Foundation AI & Data.

A policy is only as good as the variety it has seen. Simulation is how you buy that variety, which means thousands of worlds with different friction, mass, lighting, and object placement all stepping at once. Fanning out over randomized environments is plain async, and the cluster scales back to zero between experiments.

• Concurrency you can cap. `flyte.map.aio(fn, worlds, concurrency=256)` bounds how many environments run at once, for a randomization sweep wider than the cluster.
• Isaac Sim, Isaac Lab, MuJoCo, or your own. The simulator is a pip install in a task image, so a GPU-accelerated scene and a lightweight contact model can run in the same pipeline.
• Built for spot. `interruptible=True` puts simulation on spot GPUs and cuts compute costs by more than 90%. Losing a world is expected here, not an incident.

Scene complexity decides your GPU, and you find that out at runtime. A flat plane with one arm steps in a fraction of the memory a cluttered bin with deformable objects and full contact physics needs. Union hands you a typed error instead of a stack trace, so the scene that blows memory comes back as an exception you can catch and re-run on a bigger box.

• Typed infrastructure errors. `OOMError`, `TaskInterruptedError`, `TaskTimeoutError`, `ImagePullBackOffError`. Failures you can branch on.
• Resources changed at runtime. `.override(resources=...)` re-runs the same scene with more memory, or moves it from an L40S to an H100.
• Three tiers, one workflow. GPU simulation, GPU policy updates, and CPU work for video encoding and dataset assembly. Each task declares what it needs instead of the pipeline sizing everything for its heaviest step.
• A wedged simulator is a typed error too. `TaskTimeoutError` catches the physics step that stopped converging, so one bad randomization does not stall the iteration behind it.

Locomotion policies are measured in days, not minutes. A gait that took four days and twenty thousand iterations to stop falling over is gone if the run dies at nineteen thousand. Union records what finished as the run happens, outside the node doing the work, and that record is what you resume from.

• Recover what failed. Point `recover` at a prior run. Finished iterations are reused and only what failed or changed runs again, even without caching enabled.
• Checkpoints that survive the node. `flyte.Checkpoint` saves policy and optimizer state each iteration, so a preemption resumes at the iteration it stopped rather than at a fresh initialization.
• Fork the randomization, keep the policy. `flyte.rerun` branches a prior run at the iteration you pick, so a wider mass range or a new terrain curriculum starts from a trained policy instead of from scratch.

Sim produces more than policies. A run also leaves behind scene assets, generated demonstrations, and evaluation episodes, and all of it usually lands as paths one engineer remembers. Artifacts are typed, versioned values that persist past the run that made them, so a hardware test or an imitation learning job consumes what sim produced without re-running it.

• Passed between workflows and apps. A simulation run's demonstration set is a training workflow's typed input, without re-running the producer.
• Versioned, not overwritten. A new version is a new artifact, so a policy that regressed on hardware names the exact scene assets and checkpoint it shipped with.
• Events, not polling. `flyte.OnArtifact` fires a hardware-in-the-loop eval whenever a new policy is promoted, with no cron hacks in between.

The sim-to-real gap is a debugging problem before it is a research problem. When a policy works in simulation and fails on the robot, the question is what the simulation actually was, and a seed alone will not answer it. Every run keeps what it takes to reproduce a result: the code that executed, the infrastructure it ran on, and the configuration applied to both.

• Inputs and outputs per run. Open a training run from months ago and see the exact randomization ranges, reward terms, and robot description behind a policy.
• The code that ran, not the code today. Each run resolves to its own code bundle and container image, down to the simulator build and the physics engine version.
• Full lineage from scene to policy. Every checkpoint traces back through its demonstrations and scene assets to the URDF and mesh versions it trained against.