New in Flyte 2: patterns that weren’t possible in Flyte 1

Flyte 1 was a batch orchestration system: everything ran as a finite DAG that started, did work, and finished. Flyte 2 keeps all of that and adds long-running services, high-throughput batch inference, and sandboxed code execution — so the same project that trains your model can also serve it, host a dashboard, saturate a GPU, or safely run LLM-generated code. There is no v1 counterpart to migrate here; these are net-new capabilities.

Real-time inference and model serving

Instead of scoring a batch and exiting, you can stand up an always-on REST endpoint from a FastAPIAppEnvironment and deploy it with flyte.deploy. The app can load a model artifact produced by one of your training tasks.

ml_model_serving.py 
app = FastAPI(title="ML Model API")

# Define request/response models
class PredictionRequest(BaseModel):
    feature1: float
    feature2: float
    feature3: float

class PredictionResponse(BaseModel):
    prediction: float
    probability: float

# Load model (you would typically load this from storage)
model = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    global model
    model_path = os.getenv("MODEL_PATH", "/app/models/model.joblib")
    # In production, load from your storage
    if os.path.exists(model_path):
        with open(model_path, "rb") as f:
            model = joblib.load(f)
    yield

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    # Make prediction
    # prediction = model.predict([[request.feature1, request.feature2, request.feature3]])

    # Dummy prediction for demo
    prediction = 0.85
    probability = 0.92

    return PredictionResponse(
        prediction=prediction,
        probability=probability,
    )

env = FastAPIAppEnvironment(
    name="ml-model-api",
    app=app,
    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(
        "fastapi",
        "uvicorn",
        "scikit-learn",
        "pydantic",
        "joblib",
    ),
    parameters=[
        flyte.app.Parameter(
            name="model_file",
            value=flyte.io.File.from_existing_remote("s3://bucket/models/model.joblib"),
            mount="/app/models",
            env_var="MODEL_PATH",
        ),
    ],
    resources=flyte.Resources(cpu=2, memory="2Gi"),
    requires_auth=False,
)

See Build apps and Serve and deploy apps.

LLM serving

For serving large language models, the flyteplugins-vllm integration gives you a production-grade vLLM server (with autoscaling to zero) in a few lines.

basic_vllm.py 
vllm_app = VLLMAppEnvironment(
    name="my-llm-app",
    model_hf_path="Qwen/Qwen3-0.6B",  # HuggingFace model path
    model_id="qwen3-0.6b",  # Model ID exposed by vLLM
    resources=flyte.Resources(
        cpu="4",
        memory="16Gi",
        gpu="L40s:1",  # GPU required for LLM serving
        disk="10Gi",
    ),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),
        scaledown_after=300,  # Scale down after 5 minutes of inactivity
    ),
    requires_auth=False,
)

See vLLM and the other Native app integrations (SGLang, Streamlit, FastAPI).

App serving (dashboards and APIs)

Any web app — a Streamlit dashboard for exploring results, a Gradio demo, a Flask backend — runs as an AppEnvironment. Configure the image, resources, port, autoscaling, and a custom subdomain, then flyte.serve it.

hello-world-app.py 
app_env = flyte.app.AppEnvironment(
    name="hello-world-app",
    image=image,
    args=["streamlit", "hello", "--server.port", "8080"],
    port=8080,
    resources=flyte.Resources(cpu="1", memory="1Gi"),
    requires_auth=False,
    domain=flyte.app.Domain(subdomain="hello"),
)

See Introducing apps and Configure apps.

Dynamic batching for GPU inference

For in-process batch inference, DynamicBatcher from flyte.extras keeps an expensive GPU saturated: async producers load and preprocess data concurrently while a single consumer feeds the model in optimally-sized batches, with built-in backpressure. This replaces the Flyte 1 pattern of standing up a separate inference server just to get request batching.

import asyncio
from flyte.extras import DynamicBatcher

async with DynamicBatcher(
    process_fn=run_inference,   # takes a batch, returns results in the same order
    target_batch_cost=1000,     # cost budget per batch
    max_batch_size=64,          # hard cap on records per batch
    batch_timeout_s=0.05,       # max wait before dispatching a partial batch
) as batcher:
    futures = [await batcher.submit(record) for record in records]
    results = await asyncio.gather(*futures)

submit() is non-blocking and returns a Future; when the queue is full it applies backpressure automatically. See Batch inference, which also covers TokenBatcher for token-aware LLM batching.

Sandboxed code execution

flyte.sandbox.create() runs arbitrary Python code or shell commands inside an ephemeral, single-use Docker container — built on demand from declared dependencies, executed once, then discarded. Only declared inputs go in and only declared outputs come back, which makes it the safe way to run untrusted code, most importantly code generated by an LLM.

code_sandbox.py 
# sandbox_environment provides the base runtime for code sandboxes.
# Include it in depends_on so the sandbox runtime is available when tasks execute.
env = flyte.TaskEnvironment(
    name="sandbox-demo",
    image=flyte.Image.from_debian_base(name="sandbox-demo"),
    depends_on=[sandbox_environment],
)

# Auto-IO mode: pure computation. The code string runs in an isolated sandbox;
# only the declared inputs go in and only the declared outputs come back.
sum_sandbox = flyte.sandbox.create(
    name="sum-to-n",
    code="total = sum(range(n + 1)) if conditional else 0",
    inputs={"n": int, "conditional": bool},
    outputs={"total": int},
)

Call it from a task with await sum_sandbox.run.aio(n=10, conditional=True). See Code sandboxing. This also powers code mode (programmatic tool calling), where an agent writes a whole program instead of emitting one tool call at a time — see Programmatic tool calling for agents.

Next