Union Artifacts part 2: Building durable, event-driven AI systems

“No use to them,” said Alice; “but it’s useful to the people that name them, I suppose. If not, why do things have names at all?”
— Lewis Carroll, Through the Looking-Glass

This is part 2 of the Union Artifacts blog series. See part 1.

A durable run already keeps its outputs. Every file, directory, and dataframe a task returns is offloaded to object storage, and a recovered or replayed run finds them where it left them. The bytes survive the pod, the node, and the retry.

What doesn't survive is knowing where they are. The output lives at a path with a run ID and an action ID in it, and the only durable record of that path is whatever someone copied it into: a Slack thread, a serving config, a notebook cell. The data is durable, and the address is only as durable as the person who remembers it. Rerun the pipeline and there's a second path. Is the consumer on the right one?

The usual fix is a second system, a model registry or a data catalog next to the orchestrator, and now two records of one event have to agree. Artifacts take the other route. The system that ran the task gives its output a name, a version, and a record of where it came from, and every consumer refers to the name. Two more things follow once outputs have names. Data that arrives in slices, a day at a time or a region at a time, needs the slice to be part of the name. And a pipeline that waits for someone else's data needs to hear when it lands, rather than guess.

This post covers all three: artifacts, partitions, and triggers.

"Artifacts is where our teams manage and share the datasets and models they produce. Every artifact carries the labels our pipelines set, thousands of values on some keys, and we can still find the exact clip we need by key, value, date, and who produced it. Collaborating with Union has been amazing: we asked for scoped filters on a Wednesday and they were live on Thursday."
— Eryk Gorzynski, Third Dimension

Durable data needs a durable name

An artifact is a named, versioned reference to an object your tasks already wrote. Four properties, each of which rules something out:

• It's a reference, not a copy. The bytes stay where the task put them. Registering an artifact doesn't move or duplicate data, so there's no second storage bill and no sync to fall behind.
• It's offloaded. Only a `flyte.io.File`, `flyte.io.Dir`, or `flyte.io.DataFrame` can be an artifact. Primitives, dataclasses, and Pydantic models cannot. An artifact is an addressable asset, not a scalar.
• It's versioned by production. Registering under the same name again creates a new version. You don't bump anything; running the task again is the version bump.
• It's append-only. A new version never overwrites an old one. A read by name returns the newest version, and every older one stays addressable by its version ID. This is the durability property that matters: the thing a consumer pinned last month is still there.

Registering a task output

The most common way to create an artifact is to wrap a task's return value. Two things are required, the flag and the wrapper:

import flyte
from flyte import artifacts
from flyte.io import File

env = flyte.TaskEnvironment(name="training")

@env.task(produces_artifacts=True)
async def train() -> File:
    weights = await File.from_local("model.pt")
    return artifacts.new(
        weights,
        artifacts.Metadata(
            name="trained-model",
            description="Classifier trained on the latest training set",
            attrs={"framework": "pytorch"},
        ),
    )

`artifacts.new()` returns a zero-copy wrapper that behaves exactly like the original `File`, so the task's declared return type doesn't change and neither does anything downstream. `produces_artifacts=True` is what tells the platform to publish on the way out.

The failure mode is worth knowing. Leave the flag off and the task still succeeds, still returns a perfectly good `File`, and registers nothing: no artifact, no version, no trigger fires. Nothing warns you. If an artifact never showed up, check the decorator first.

`Metadata` carries identity. Only `name` is required; omit `version` and it comes from the producing run. `kind` (`"model"`, `"data"`, or `"generic"`) makes the registry queryable, a card renders a model or data card in the UI, and free-form string `attrs` become filters later.

Partitions: one name, many slices

A name per artifact works for a model. It breaks down for data that arrives in slices. An ingest job writes raw events every day for each region. If every run publishes a new version of one `raw_events` artifact, the versions pile up in a flat list. Finding "the US data for August 2" means reading descriptions, and a backfill has no way to ask which days already exist.

Partitions make the slice part of the identity:

import asyncio
from datetime import date, timedelta

import flyte
import flyte.artifacts as artifacts
from flyte.io import File

env = flyte.TaskEnvironment(name="ingest")


@env.task(produces_artifacts=True)
async def ingest(day: date, region: str) -> File:
    file = await File.from_local(f"/data/{region}/{day}.parquet")
    return artifacts.new(
        file,
        artifacts.Metadata(name="raw_events", partitions={"date": day, "region": region}),
    )


@env.task
async def main(days: int = 3) -> None:
    start = date(2026, 8, 1)
    await asyncio.gather(
        *(ingest(start + timedelta(days=d), r) for d in range(days) for r in ("us", "eu"))
    )

One run of `main` fills six partitions. The Python type of each value decides what kind of partition it is: a `date` is a daily time partition, a `datetime` an hourly one (floored to the hour in UTC), `artifacts.TimePartition(value, "week")` or `"month"` the coarser ones, and anything else a string partition. A version carries at most one time partition plus any number of string partitions. A `None` value is rejected, because an unset partition is almost always a bug.

Partitions inherit the append-only rule. Publishing to a partition that already has a version adds a version; it doesn't overwrite. A rerun of a failed day is safe, reads by partition return the newest version, and you can always see what the partition held before.

Partition keys are static. The first partitioned version of an artifact sets its keys, here `date` and `region`, and from then on they can't be changed. When several teams or jobs write the same artifact, declare the keys up front, so the first writer can't set the wrong ones by accident:

from datetime import date, datetime
from flyte.remote import Artifact

Artifact.declare("raw_events", {"date": date, "region": str})
Artifact.declare("raw_events_hourly", {"hour": datetime, "region": str})
Artifact.declare("monthly_report", {"date": "month"})

Declaring the same keys again is a no-op, and declaring different ones fails. If you need different partitions, publish under a new artifact name.

Asking what you have

Different consumers ask different questions of the same grid, and each is one call:

from datetime import date
from flyte.remote import Artifact

# one cell: the newest version of that partition
us_aug_2 = Artifact.get("raw_events", date=date(2026, 8, 2), region="us")

# a range: one version per partition, not every version
august = list(
    Artifact.listall(
        "raw_events",
        date=(date(2026, 8, 1), date(2026, 8, 31)),
        region=["us", "eu"],
        latest_per_partition=True,
    )
)

# which days have EU data: the distinct values of "date" among the eu versions
Artifact.partition_values("raw_events", "date", region="eu")
# -> [date(2026, 8, 1), date(2026, 8, 2), date(2026, 8, 3), date(2026, 8, 5), ...]

The range is inclusive at both ends, and a list on a string key matches any of its values.

`partition_values` doesn't return artifacts. It returns the distinct values one partition key takes across an artifact's versions, sorted, and narrowed by any other keys you pass. Here that's every day that has an EU partition, so the missing August 4 shows up as a hole in the list. That's the query a backfill plans with: compare the days that came back against the days you expect, and fill the gaps. Swap the keys and you get the other view: `partition_values("raw_events", "region", date=date(2026, 8, 4))` is every region that has data for that day.

The CLI takes the same filters:

flyte get artifact raw_events --partition date=2026-08-01..2026-08-31 --partition region=us,eu --latest-per-partition

Publishing outputs of tasks you don't own

Partitioned data usually passes through tasks someone else wrote. Another team maintains a shared `clean` task that returns a plain `File` and knows nothing about artifacts. You want each call's output registered as a partition of `clean_events`, and you can't edit the task to wrap its return value.

`artifacts.produces` lets the caller declare which outputs become artifacts:

async def clean_one(day: date, region: str) -> File:
    with artifacts.produces(
        o0=artifacts.Metadata(name="clean_events", partitions={"date": day, "region": region})
    ):
        return await clean.override(produces_artifacts=True)(day=day, region=region)

`o0` is the first output slot, `o1` the second; slots you don't name stay ordinary outputs. The call still needs `produces_artifacts=True`, turned on here with `.override`. Publishing happens only if the called task succeeds, and the version's source is the action that ran `clean`, so it shows up in lineage and fires triggers like any other. Keep one call per `with` block. Concurrent calls, as under `asyncio.gather`, each keep their own declaration. The same works for a remote task reference fetched with `flyte.remote.Task.get`.

The same partitions work from outside a task, which is how you backfill from data you already have:

Artifact.create(
    File.from_local_sync("backfill/2026-07-31-us.parquet"),
    name="raw_events",
    partitions={"date": date(2026, 7, 31), "region": "us"},
)

Why triggers: a schedule guesses, an event knows

Once data has a name, the next question is when to act on it. The default answer in most pipelines is a schedule. The upstream drop usually lands before 06:00, so the downstream job runs at 06:00.

A schedule encodes a guess about someone else's system. When the guess is right, you still wait for the clock after the data is ready. When it's wrong, the job runs on yesterday's data or fails, and someone reruns it by hand. Padding the schedule trades wrong runs for late ones. A sensor that polls trades them for a loop that burns compute asking. And the guess breaks entirely the day the upstream team moves their job, because nothing tells you.

An artifact trigger replaces the guess with the event. Publishing a version is a fact the platform records, so the platform can start the downstream run at that moment, with that version bound to its input. The consumer doesn't need to know when the producer runs, or who the producer is.

Triggering on new versions

A trigger is a `flyte.Trigger` whose automation is `flyte.OnArtifact`. The `flyte.TriggeredArtifact` sentinel marks which input receives the new version:

retrain = flyte.Trigger(
    name="retrain-on-new-model",
    automation=flyte.OnArtifact(name="customer_model"),
    inputs={"model": flyte.TriggeredArtifact, "threshold": 0.5},
    description="Validate every new customer_model version",
)

@env.task(triggers=(retrain,))
async def validate(model: File, threshold: float = 0.5) -> str:
    ...

`flyte deploy validation.py env` registers it. From then on, every new version of `customer_model` in the task's project and domain starts a run of `validate`. At most one input can be `TriggeredArtifact`, and every other input needs a value in the trigger or a default on the task. Pass `version=` to `OnArtifact` to fire only when one exact version is published.

The property that makes triggers a handoff point is that they don't care who published. A task output, a caller's `produces` declaration, an `Artifact.create()` from a CI job, a CLI upload, a Hugging Face prefetch: each is a new version, and each fires the trigger. A partner drops a dataset, something publishes it, your pipeline starts. Nobody wrote an integration.

Triggering on partitions

A plain artifact trigger fires on every version. For partitioned data that's usually too much. The US team's cleaning pipeline should run when a US partition of `raw_events` lands, not an EU one, and it needs to know which day just arrived without opening the file.

Scope the trigger to a partition and bind the partition values to inputs:

from datetime import datetime

process_us = flyte.Trigger(
    name="process-new-us-partition",
    automation=flyte.OnArtifact(name="raw_events", region="us"),
    inputs={
        "events": flyte.TriggeredArtifact,
        "day": flyte.TriggeredPartition("date"),
        "region": flyte.TriggeredPartition("region"),
    },
    description="Process every new US partition of raw_events",
)


@env.task(triggers=(process_us,))
async def process_partition(events: File, day: datetime, region: str) -> str:
    ...

A version fires the trigger only if its partitions carry every key and value you list. Only string partitions narrow a trigger. A time partition arrives as an input instead: a `datetime` at the start of the partition in UTC, so a daily partition arrives as midnight UTC. To act on particular days, bind the day and check it in the task. A version published for `region="eu"` is still registered, but it fires nothing here.

A trigger can also be fired by hand, from the UI or with `flyte.run(trigger, ...)`. No version is being published in that case, so pass the artifact and the partition-bound inputs yourself as keyword overrides. That's also how you replay one day through a pipeline after fixing a bug.

Apps: pinned at deploy

The last consumer is the one serving traffic. `flyte.app.ArtifactValue` mounts weights into a serving container instead of baking them into the image:

env = flyte.app.AppEnvironment(
    name="classifier-api",
    parameters=[
        flyte.app.Parameter(
            name="model",
            value=flyte.app.ArtifactValue(name="trained-model", type="directory"),
            mount="/tmp/model",
        )
    ],
)

Versions are pinned at deploy time. Leave out `version` and the latest at deploy is chosen and frozen, so a new version doesn't swap weights under a running app. That's deliberate: triggers are for pipelines that should react to every version, and pinning is for services that shouldn't. Redeploy to move.

Putting it together

Put the pieces in a row and the pipeline stops being one DAG someone has to own end to end. `ingest` publishes partitions of `raw_events`. A partition trigger starts the cleaning run for each new US day, which calls another team's `clean` and publishes its output as `clean_events` with `produces`. A trigger on `clean_events` starts training, which publishes `customer_model`. A trigger validates each new model, and the serving app pins the one that passed. No stage calls the next one. Each stage knows the name it reads and the name it writes, and any stage can be rerun, replaced, or backfilled without touching the others.

Lineage, and what it actually covers

Because the same system runs the task, publishes the artifact, and fires the trigger, lineage is a byproduct rather than an integration. Each version records how it came to exist: a task output records the producing run and action, with a link to its inputs and logs; an upload records who published it and the `external_ref`; a prefetch records the source repo and commit. On the consumer side, Union records the triggers watching the name and the runs they started, and the apps whose deployment resolved the artifact, down to the pinned version.

Lineage today is not universal, and the gaps are specific. Calling `Artifact.get()` in your own code and reading the data is a plain fetch and creates no consumer edge. Passing an artifact directly as a task input doesn't record one either, not yet. The producer side is complete; the consumer side covers triggers and apps and stops there. That's enough to answer the two questions people actually page each other about: what does this model come from, and what breaks if I change this dataset.

flyte get artifact --source-run my_run
flyte get artifact raw_events --partition region=us --latest-per-partition
flyte get artifact --kind model --attr framework=torch

Recap

Durability: an artifact is a durable name for durable data. Versions are append-only, so what a consumer pinned stays addressable.

Registration: `produces_artifacts=True` plus `artifacts.new()`, or `artifacts.produces(...)` from the caller for a task you don't own. Without the flag, nothing is published and nothing tells you.

Partitions: `Metadata(partitions={...})` makes the slice part of the identity. Keys are static: set by the first partitioned version or by `Artifact.declare`, and changed only by moving to a new name. Republishing a partition adds a version.

Retrieval: `Artifact.get` for one version or one partition, `listall` with ranges and `latest_per_partition` for many, `partition_values` to list the values a key has and find the gaps.

Triggers: `OnArtifact` fires on every new version, whoever published it. Narrow it with string partitions, and bind partition values with `TriggeredPartition`.

Apps: `ArtifactValue` pins at deploy. Pipelines react, services don't.

None of this is a second system. It's the orchestrator writing down what it already knew, and telling the next stage when it happened.

Try it

Partitions, partition triggers, `Artifact.declare`, and `artifacts.produces` need flyte 2.10.0 or later.

# pip install "flyte>=2.10.0"
from datetime import date

import flyte
from flyte.io import File
from flyte.remote import Artifact

flyte.init_from_config()
Artifact.create(
    File.from_local_sync("day1.parquet"),
    name="raw_events",
    partitions={"date": date(2026, 8, 1), "region": "us"},
)

flyte get artifact raw_events --partition region=us

Then deploy a partition trigger on `region=us` and publish an `eu` and a `us` partition to watch only one of them fire. The Artifacts guide covers the full API, Partitioned artifacts and Trigger on partitions go deeper on this post's new parts, and Automatic lineage tracking documents the current edges, including the ones that aren't drawn yet.