FastAPI

FastAPI gives you full control over an endpoint’s routes, request schema, and pre- and post-processing. Use it for models an LLM server doesn’t cover, such as classifiers, embedding models, and scikit-learn or PyTorch models, or when an LLM needs custom logic around it.

Flyte’s FastAPIAppEnvironment runs a FastAPI app with Uvicorn on your compute, so you only write the app. Unlike the vLLM, SGLang, Ollama, and llama.cpp endpoints, this endpoint doesn’t expose the OpenAI API unless you implement those routes yourself.

Define the API

This example serves a sentiment classifier from Hugging Face. The lifespan handler loads the model once per replica, before the app accepts traffic:

fastapi_endpoint.py
                
from contextlib import asynccontextmanager

import flyte
import flyte.app
from fastapi import FastAPI
from flyte.app.extras import FastAPIAppEnvironment
from pydantic import BaseModel
from transformers import pipeline

MODEL = "distilbert-base-uncased-finetuned-sst-2-english"
state = {}

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load the model once per replica, before the app accepts traffic.
    state["classifier"] = pipeline("text-classification", model=MODEL)
    yield

app = FastAPI(title="Sentiment endpoint", lifespan=lifespan)

class PredictRequest(BaseModel):
    text: str

class PredictResponse(BaseModel):
    label: str
    score: float

@app.post("/predict")
def predict(request: PredictRequest) -> PredictResponse:
    return PredictResponse(**state["classifier"](request.text)[0])

predict is a plain def, so FastAPI runs it in a thread pool and the blocking model call doesn’t hold up the event loop.

Define the app environment

Pass the FastAPI app to the environment, along with the image and the resources the model needs:

fastapi_endpoint.py
                
app_env = FastAPIAppEnvironment(
    name="fastapi-endpoint",
    app=app,
    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(
        "fastapi", "uvicorn", "torch", "transformers"
    ),
    resources=flyte.Resources(cpu="2", memory="4Gi", disk="10Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 2), scaledown_after=300),
)

To run on a GPU, add a gpu value to resources and pass device=0 to pipeline.

Deploy

fastapi_endpoint.py
                
if __name__ == "__main__":
    flyte.init_from_config(root_dir=Path(__file__).parent)
    deployment = flyte.serve(app_env)
    print(f"Deployed: {deployment.url}")
    print(f"API docs: {deployment.url}/docs")

Call the endpoint

With requires_auth=True, the default, callers authenticate with an API key:

        
    
curl -X POST "https://your-app-url/predict" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FLYTE_API_KEY" \
  -d '{"text": "I love this"}'

FastAPI also generates interactive API docs at /docs. For app-managed authentication, see Secret-based authentication.

Next steps

Load your own weights instead of a Hugging Face model: Deploy trained models
Avoid downloading weights on every cold start: Prefetching models
Tune replicas and scale-to-zero: Autoscaling apps
Use a dedicated serving framework: Other frameworks