# New in Flyte 2: patterns that weren't possible in Flyte 1

Flyte 1 was a batch orchestration system: everything ran as a finite DAG that started, did work, and finished. Flyte 2 keeps all of that and adds **long-running services**, **high-throughput batch inference**, and **sandboxed code execution** — so the same project that trains your model can also serve it, host a dashboard, saturate a GPU, or safely run LLM-generated code. There is no v1 counterpart to migrate here; these are net-new capabilities.

## Real-time inference and model serving

Instead of scoring a batch and exiting, you can stand up an always-on REST endpoint from a `FastAPIAppEnvironment` and deploy it with `flyte.deploy`. The app can load a model artifact produced by one of your training tasks.

```
app = FastAPI(title="ML Model API")

# Define request/response models
class PredictionRequest(BaseModel):
    feature1: float
    feature2: float
    feature3: float

class PredictionResponse(BaseModel):
    prediction: float
    probability: float

# Load model (you would typically load this from storage)
model = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    global model
    model_path = os.getenv("MODEL_PATH", "/app/models/model.joblib")
    # In production, load from your storage
    if os.path.exists(model_path):
        with open(model_path, "rb") as f:
            model = joblib.load(f)
    yield

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    # Make prediction
    # prediction = model.predict([[request.feature1, request.feature2, request.feature3]])

    # Dummy prediction for demo
    prediction = 0.85
    probability = 0.92

    return PredictionResponse(
        prediction=prediction,
        probability=probability,
    )

env = FastAPIAppEnvironment(
    name="ml-model-api",
    app=app,
    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(
        "fastapi",
        "uvicorn",
        "scikit-learn",
        "pydantic",
        "joblib",
    ),
    parameters=[
        flyte.app.Parameter(
            name="model_file",
            value=flyte.io.File.from_existing_remote("s3://bucket/models/model.joblib"),
            mount="/app/models",
            env_var="MODEL_PATH",
        ),
    ],
    resources=flyte.Resources(cpu=2, memory="2Gi"),
    requires_auth=False,
)
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/fastapi/ml_model_serving.py*

See [Build apps](https://www.union.ai/docs/v2/union/user-guide/migration/build-apps/_index) and [Serve and deploy apps](https://www.union.ai/docs/v2/union/user-guide/migration/serve-and-deploy-apps/_index).

## LLM serving

For serving large language models, the `flyteplugins-vllm` integration gives you a production-grade vLLM server (with autoscaling to zero) in a few lines.

```
vllm_app = VLLMAppEnvironment(
    name="my-llm-app",
    model_hf_path="Qwen/Qwen3-0.6B",  # HuggingFace model path
    model_id="qwen3-0.6b",  # Model ID exposed by vLLM
    resources=flyte.Resources(
        cpu="4",
        memory="16Gi",
        gpu="L40s:1",  # GPU required for LLM serving
        disk="10Gi",
    ),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),
        scaledown_after=300,  # Scale down after 5 minutes of inactivity
    ),
    requires_auth=False,
)
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/vllm/basic_vllm.py*

See [vLLM](https://www.union.ai/docs/v2/union/user-guide/migration/native-app-integrations/vllm-app) and the other [Native app integrations](https://www.union.ai/docs/v2/union/user-guide/migration/native-app-integrations/_index) (SGLang, Streamlit, FastAPI).

## App serving (dashboards and APIs)

Any web app — a Streamlit dashboard for exploring results, a Gradio demo, a Flask backend — runs as an `AppEnvironment`. Configure the image, resources, port, autoscaling, and a custom subdomain, then `flyte.serve` it.

```
app_env = flyte.app.AppEnvironment(
    name="hello-world-app",
    image=image,
    args=["streamlit", "hello", "--server.port", "8080"],
    port=8080,
    resources=flyte.Resources(cpu="1", memory="1Gi"),
    requires_auth=False,
    domain=flyte.app.Domain(subdomain="hello"),
)
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/configure-apps/hello-world-app.py*

See [Introducing apps](https://www.union.ai/docs/v2/union/user-guide/migration/core-concepts/introducing-apps) and [Configure apps](https://www.union.ai/docs/v2/union/user-guide/migration/configure-apps/_index).

## Dynamic batching for GPU inference

For in-process batch inference, `DynamicBatcher` from `flyte.extras` keeps an expensive GPU saturated: async producers load and preprocess data concurrently while a single consumer feeds the model in optimally-sized batches, with built-in backpressure. This replaces the Flyte 1 pattern of standing up a separate inference server just to get request batching.

```python
import asyncio
from flyte.extras import DynamicBatcher

async with DynamicBatcher(
    process_fn=run_inference,   # takes a batch, returns results in the same order
    target_batch_cost=1000,     # cost budget per batch
    max_batch_size=64,          # hard cap on records per batch
    batch_timeout_s=0.05,       # max wait before dispatching a partial batch
) as batcher:
    futures = [await batcher.submit(record) for record in records]
    results = await asyncio.gather(*futures)
```

`submit()` is non-blocking and returns a `Future`; when the queue is full it applies backpressure automatically. See [Batch inference](https://www.union.ai/docs/v2/union/user-guide/migration/run-scaling/batch-inference), which also covers `TokenBatcher` for token-aware LLM batching.

## Sandboxed code execution

`flyte.sandbox.create()` runs arbitrary Python code or shell commands inside an ephemeral, single-use Docker container — built on demand from declared dependencies, executed once, then discarded. Only declared inputs go in and only declared outputs come back, which makes it the safe way to run **untrusted code, most importantly code generated by an LLM**.

```
# sandbox_environment provides the base runtime for code sandboxes.
# Include it in depends_on so the sandbox runtime is available when tasks execute.
env = flyte.TaskEnvironment(
    name="sandbox-demo",
    image=flyte.Image.from_debian_base(name="sandbox-demo"),
    depends_on=[sandbox_environment],
)

# Auto-IO mode: pure computation. The code string runs in an isolated sandbox;
# only the declared inputs go in and only the declared outputs come back.
sum_sandbox = flyte.sandbox.create(
    name="sum-to-n",
    code="total = sum(range(n + 1)) if conditional else 0",
    inputs={"n": int, "conditional": bool},
    outputs={"total": int},
)
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/sandboxing/code_sandbox.py*

Call it from a task with `await sum_sandbox.run.aio(n=10, conditional=True)`. See [Code sandboxing](https://www.union.ai/docs/v2/union/user-guide/migration/sandboxing/code-sandboxing). This also powers **code mode** (programmatic tool calling), where an agent writes a whole program instead of emitting one tool call at a time — see [Programmatic tool calling for agents](https://www.union.ai/docs/v2/union/user-guide/migration/sandboxing/code-mode).

## Next

- [Gotchas and caveats](https://www.union.ai/docs/v2/union/user-guide/migration/flyte-2/new-in-flyte-2/gotchas-and-caveats) — caveats of the new execution model
- [Hybrid v1 and v2 pipelines](https://www.union.ai/docs/v2/union/user-guide/migration/flyte-2/new-in-flyte-2/hybrid-pipelines) — calling between v1 and v2 during the transition

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/user-guide/migration/flyte-2/new-in-flyte-2.md
**HTML**: https://www.union.ai/docs/v2/union/user-guide/migration/flyte-2/new-in-flyte-2/