From fleet logs to production models. At AV scale.

Union is built on Flyte, the open-source AI runtime we create and maintain under the Linux Foundation AI & Data.

Every drive can produce terabytes of multimodal data across camera, LiDAR, radar, and IMU. Fanning out over tens of thousands of drives is plain async, so labeling, embedding, and scene curation keep up with the fleet instead of falling behind it.

• Concurrency you can cap. `flyte.map.aio(fn, drives, concurrency=500)` bounds how many run at once, for a fan-out wider than the cluster.
• Retries with pacing. `retries=5`, or a `RetryStrategy` with exponential backoff so a flaky data source gets time to come back.
• Built for spot. `interruptible=True` puts perception work on spot GPUs, cutting pipeline compute costs at fleet scale. Losing one is expected here, not an incident.

When a perception or planning model changes, you replay it against years of historical drives to prove it got better, not worse. A resim that dies eighty percent through and restarts from the top pays the GPU bill twice. Union records what finished as the run happens, outside the node doing the work, and that record is what you resume from.

• Recover what failed. Point `recover` at a prior run. Scored drives are reused and only what failed or changed runs again, even without caching enabled.
• Fork from any step. `flyte.rerun` starts a prior resim again at the step you pick, with a new model version or a different eval set.
• Replay with today's code. `--rerun-from` runs a prior run's inputs against the code on your machine right now.

ISO 26262, UNECE R157, and internal safety cases all ask the same question: what data, what code, and what parameters produced this model. Every run keeps what it takes to reproduce a result: the code that executed, the infrastructure it ran on, and the configuration applied to both. That is the chain of custody a release review needs, back to the sensor data the model was trained on.

• Inputs and outputs per run. Open an execution from months ago and see exactly which drives went in and which metrics came out.
• The code that ran, not the code today. Each run resolves to its own code bundle and container image, for exact dependency reproduction years later.
• Full lineage from raw logs to shipped model. Every model traces back through resim metrics and curated datasets to the sensor logs behind it.

Scenes are not the same size. A highway drive at night decodes in a fraction of the memory a dense urban intersection needs with every camera, LiDAR sweep, and radar return loaded at once. Union hands you a typed error instead of a stack trace, so the scene that blows memory comes back as an exception you can catch and re-run on a bigger box.

• Typed infrastructure errors. `OOMError`, `TaskInterruptedError`, `TaskTimeoutError`,` ImagePullBackOffError`. Failures you can branch on.
• Resources changed at runtime. `.override(resources=...)` re-runs the same scene with different memory, CPU, or accelerator.
• Two tiers, one workflow. Cheap CPU workers curate drives, spot A100s run perception, and each task declares its own image and hardware.
• The handler does not have to be human. Provisioning is an ordinary Python call, so the same API you write a retry policy against is one an agent can call while a run is in flight.

A trained model should not be a checkpoint path in a bucket that one engineer remembers. Artifacts are typed, versioned values that persist past the run that made them, so a resim consumes a candidate model without re-running the training that produced it, and a new model version can fire the eval by itself.

• Passed between workflows and apps. One pipeline's model or curated dataset is another's typed input, without re-running the producer.
• Versioned, not overwritten. A new version is a new artifact, so a resim run resolves to the exact model and drives it scored.
• Events, not polling. `flyte.OnArtifact` fires a run whenever a new model version is registered, with no cron hacks in between.