Reuse Deployed Tasks Across Teams

AI engineering tip of the week: Reuse Deployed Tasks Across Teams

Your data team has a Spark task that cleans and transforms raw data. Your ML team needs that cleaned data for training. In most setups, you'd either copy the code, create a shared library, or build some API layer in between.

With Flyte remote tasks, you just reference the deployed task by name. No copied code, no shared library, no API glue layer in between.

Reference a deployed task

import flyte
import flyte.remote

# Get a reference to a task someone else deployed
data_processor = flyte.remote.Task.get(
    "data_team.clean_dataset",
    auto_version="latest",
)

env = flyte.TaskEnvironment(name="ml_pipeline")

@env.task
async def train_model() -> float:
    # Call the remote task just like a local one
    clean_data = await data_processor(raw_path="s3://bucket/raw_data.csv")
    # ... train on clean_data
    return model

The `data_team.clean_dataset` task runs in its own environment with its own image and dependencies. Your ML pipeline doesn't need Spark, pandas, or anything the data team uses. It just calls the task and gets the result.

Lazy loading keeps imports fast

`flyte.remote.Task.get()` returns a lazy reference. No network call happens until you actually invoke the task. This means your module imports stay fast even if you reference dozens of remote tasks:

# These are instant, no network calls
preprocessor = flyte.remote.Task.get("data.preprocess", auto_version="latest")
embedder = flyte.remote.Task.get("nlp.embed_text", auto_version="latest")
scorer = flyte.remote.Task.get("ml.score_model", auto_version="latest")

@env.task
async def pipeline(text: str) -> float:
    # Network calls happen here when tasks are invoked
    clean = await preprocessor(text=text)
    vectors = await embedder(text=clean)
    score = await scorer(vectors=vectors)
    return score

Pin versions for production

During development, `auto_version="latest"` is convenient. For production, pin to a specific version:

# Always use this exact deployed version
data_processor = flyte.remote.Task.get(
    "data_team.clean_dataset",
    version="4f3a9c1e8b2d7a05c6e1f0b93d2a4c88",
)

The version here is the deployment version, not a run ID. By default Flyte computes it as a hash of the environment definitions, the code bundle, and the image cache, so it looks like the hex string above. If you'd rather pin to something human-readable, set it at deploy time and the version becomes whatever you passed:

# flyte deploy --version v2.1.0 clean_dataset.py data_team
data_processor = flyte.remote.Task.get("data_team.clean_dataset", version="v2.1.0")

Override resources on the fly

Need more resources than the original task was configured with? Override them:

data_processor = flyte.remote.Task.get(
    "data_team.clean_dataset",
    auto_version="latest",
)

# Run with more memory for a bigger dataset
big_processor = data_processor.override(
    resources=flyte.Resources(cpu="16", memory="64Gi"),
    retries=3,
)

@env.task
async def process_big_dataset() -> str:
    return await big_processor(raw_path="s3://bucket/huge_dataset.csv")

`override()` has to fetch the task to apply the overrides, so unlike `get()` it isn't free, it makes a network call at the point you call it. Inside an async task body, use the async form:

@env.task
async def process_big_dataset() -> str:
    big_processor = await data_processor.override.aio(
        resources=flyte.Resources(cpu="16", memory="64Gi"),
        retries=3,
    )
    return await big_processor(raw_path="s3://bucket/huge_dataset.csv")

You can override resources, retries, timeouts, environment variables, and cache settings without changing the original task.

Eagerly validate a remote task exists

If you want to catch missing tasks early (like at service startup), use .fetch():

import flyte.remote
import flyte.errors

processor = flyte.remote.Task.get("data.preprocess", auto_version="latest")

try:
    details = await processor.fetch.aio()
    print(f"Task found: {details.name}, version: {details.version}")
except flyte.errors.RemoteTaskNotFoundError:
    print("Task not deployed yet!")

Why this is powerful

• Independent release cycles: The data team can rewrite their task from scratch and your pipeline keeps working, as long as the interface holds.
• No dependency conflicts: Your orchestrator never installs Spark or PyTorch. Their image is theirs, yours is yours, so their package pins can't break your build.
• Version pinning: Pin an exact version in production, track latest in development.
• Resource flexibility: Override resources per-call without touching the deployed task.
• Composability: Build pipelines that span teams, projects, and infrastructure.

Full remote tasks docs: https://www.union.ai/docs/v2/flyte/user-guide/tasks/task-programming/remote-tasks/

See what's happening in the Flyte Community:

Latest from the blog

• Flyte 2 Is Generally Available: The Durable, Open-Source AI Runtime - Read on Union.ai
• Why Untrusted Kernel Evaluation Needs Process Isolation (and How We Built It) - Read on Union
• A Memory Store Built on Flyte and Cognee - Read on Union.ai
• Building Grounded Agents on Fresh Web Data - Read on Union.ai
• From DNA to 3D Fold: Compare a Gene Across Six Species with Carbon and ESMFold - Read on Union.ai
• Run Models, Agents and Apps on Infrastructure You Own - Read on union.ai
• Agents That Survive Production: Rebuilding 21 Design Patterns on Flyte - Read on union.ai
• Introducing Queues and Cluster Controls: Durable Workloads Under Contention - Read on union.ai

Recent talks & recordings

• Flyte 2: The Durable Runtime Built for AI - Watch on YouTube
• When the Pipeline Breaks: Building ML Infrastructure for Biotech R&D | Session 1 - Watch on YouTube
• Building Code Mode Agents - Watch on YouTube
• LLM fine-tuning with GRPO - Watch on YouTube
• LLM fine-tuning with LoRA & QLoRA - Watch on YouTube

Upcoming events

• Aug 20th: Seattle RAG & Agent Context with Vector Stores | AI Hacknight - RSVP on Luma

Releases & updates

• Flyte 2 Is Generally Available: The Durable, Open-Source AI Runtime - Read on Union.ai

<div class="button-group is-center"><a class="button" target="_blank" rel="noopener noreferrer" href="https://www.union.ai/docs/v2/flyte/user-guide/run-modes/running-devbox/">Download Devbox</a></div>

From the community

• Reinforcement Learning in MuJoCo - RSVP on Luma
• AI Book Club: Build a Reasoning Model (From Scratch) - RSVP on Luma

That's all for this week! - Sage Elliott