Other frameworks

Any framework that runs an HTTP server can serve a model on your compute. Flyte doesn’t need a dedicated integration: a generic AppEnvironment runs the server and routes traffic to it.

Every framework follows the same three steps:

Install the framework in the image, with flyte.Image and with_pip_packages.
Start the server with args, using either the framework’s CLI or a Python entrypoint.
Bind the server to 0.0.0.0 on the app’s port, so the platform can reach it. The default port is 8080.

The examples below serve the same sentiment classifier, so you can compare the frameworks directly.

Framework	Good fit for
LitServe	A small `LitAPI` class with built-in batching and GPU handling
Ray Serve	Multiple replicas or multi-model pipelines inside one app, using Ray’s programming model
BentoML	Typed service classes with generated API schemas, or an existing Bento service
Any other server	A framework you already use

LitServe

LitServe serves a LitAPI class. Start it through a --server entrypoint in the script:

litserve_endpoint.py
                
app_env = flyte.app.AppEnvironment(
    name="litserve-endpoint",
    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(
        "litserve", "torch", "transformers"
    ),
    args=["python", file_name, "--server"],
    port=PORT,
    resources=flyte.Resources(cpu="2", memory="4Gi", disk="10Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 2), scaledown_after=300),
)

The --server entrypoint starts LitServe on the app port:

litserve_endpoint.py
                
# LitServe pickles the API class into its worker processes, so define it at module level.
class Classifier(ls.LitAPI):
    def setup(self, device):
        self.classifier = pipeline("text-classification", model=MODEL, device=device)

    def decode_request(self, request):
        return request["text"]

    def predict(self, text):
        return self.classifier(text)[0]

def serve() -> None:
    server = ls.LitServer(Classifier(), accelerator="auto")
    server.run(host="0.0.0.0", port=PORT)

Deploy with flyte.serve, and run the server when the container passes --server:

litserve_endpoint.py
                
if __name__ == "__main__":
    if "--server" in sys.argv:
        serve()
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        deployment = flyte.serve(app_env)
        print(f"Deployed: {deployment.url}")

LitServe serves POST /predict by default.

Ray Serve

Ray Serve runs deployments on a Ray cluster. In this pattern, Ray runs inside a single app replica, so size resources for all of the Ray Serve replicas plus Ray’s own processes:

ray_serve_endpoint.py
                
app_env = flyte.app.AppEnvironment(
    name="ray-serve-endpoint",
    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(
        "ray[serve]", "torch", "transformers"
    ),
    args=["python", file_name, "--server"],
    port=PORT,
    resources=flyte.Resources(cpu="4", memory="8Gi", disk="10Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 1), scaledown_after=300),
)

serve.start binds the Ray Serve proxy to the app port. serve.run returns once the deployment is ready, so the entrypoint then blocks to keep the container running:

ray_serve_endpoint.py
                
def serve() -> None:
    from ray import serve as ray_serve
    from starlette.requests import Request
    from transformers import pipeline

    @ray_serve.deployment(num_replicas=2, ray_actor_options={"num_cpus": 1})
    class Classifier:
        def __init__(self):
            self.classifier = pipeline("text-classification", model=MODEL)

        async def __call__(self, request: Request) -> dict:
            body = await request.json()
            return self.classifier(body["text"])[0]

    # Bind the Ray Serve proxy to the app port on all interfaces.
    ray_serve.start(http_options={"host": "0.0.0.0", "port": PORT})
    ray_serve.run(Classifier.bind(), route_prefix="/predict")

    # `serve.run` returns once the deployment is ready. Keep the container alive.
    while True:
        time.sleep(60)

Keep the app at one replica, as in replicas=(0, 1), and scale with num_replicas in the Ray deployment. Each app replica would otherwise start its own independent Ray instance.

BentoML

BentoML defines a service as a class. Each @bentoml.api method becomes a POST route named after the method:

bentoml_endpoint.py
                
@bentoml.service(workers=1)
class Classifier:
    def __init__(self):
        from transformers import pipeline

        self.classifier = pipeline("text-classification", model=MODEL)

    @bentoml.api
    def predict(self, text: str) -> dict:
        return self.classifier(text)[0]

BentoML has its own CLI, so args can start the server directly:

bentoml_endpoint.py
                
app_env = flyte.app.AppEnvironment(
    name="bentoml-endpoint",
    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(
        "bentoml", "torch", "transformers"
    ),
    # `bentoml serve` imports the service from this module.
    args=["bentoml", "serve", f"{Path(__file__).stem}:Classifier", "--host", "0.0.0.0", "--port", str(PORT)],
    port=PORT,
    resources=flyte.Resources(cpu="2", memory="4Gi", disk="10Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 2), scaledown_after=300),
)

Any other server

The same pattern works for other servers. Point args at the command that starts the server and make sure it listens on 0.0.0.0 and the app’s port:

Framework	What it serves	Notes
MLServer	Scikit-learn, XGBoost, LightGBM, and MLflow models	Started with `mlserver start`. Set the port in `settings.json`.
MLflow	Models in the MLflow format	Started with `mlflow models serve`, which takes `--host` and `--port`.
Gradio	A browser demo with an HTTP API	Set `GRADIO_SERVER_NAME` to `0.0.0.0` and `GRADIO_SERVER_PORT` to the app port.

TorchServe is no longer maintained, and its repository was archived in August 2025. For new PyTorch endpoints, use one of the frameworks on this page or a FastAPI endpoint.

For an LLM, start with the vLLM, SGLang, Ollama, or llama.cpp endpoints.

Deploy, call, and scale

All of these run as ordinary apps, so the rest of the app documentation applies:

Authentication: requires_auth=True is the default and sits in front of the framework’s own server. See Secret-based authentication for app-managed auth.
Scaling: See Autoscaling apps.
Weights: See Prefetching models and Deploy trained models.

Troubleshooting

The app never becomes ready:

Confirm the server binds 0.0.0.0 rather than localhost. Many frameworks default to localhost or use their own default port, such as 3000 for BentoML and 8000 for Ray Serve.
Check that the server’s port matches port on the AppEnvironment.
Avoid the reserved ports 8012, 8022, 8112, and 9090.

The server starts, then the container exits:

The args command must stay in the foreground. If the entrypoint only starts a background server, as Ray Serve’s does, block afterward.

Slow first response:

The model loads at startup, so a replica scaling up from zero takes that long to respond. Raise scaledown_after, keep a minimum replica, or prefetch the weights.