Serve models on your own GPUs or CPUs, with vLLM, SGLang, Ollama, llama.cpp, FastAPI, Ray Serve, LitServe, BentoML, or another framework.
Self-hosted endpoints
Serve a model as an app on your own compute. The vLLM, SGLang, Ollama, and llama.cpp endpoints expose the OpenAI API, so code written against the OpenAI SDK reaches any of them by changing the base URL, the API key, and the model name. FastAPI and other frameworks serve any model behind the API you define.
| You want to | Use |
|---|---|
| Serve an LLM at high throughput | vLLM endpoint |
| Serve an LLM with structured generation | SGLang endpoint |
| Serve a small LLM with minimal setup | Ollama endpoint |
| Serve quantized GGUF weights, a model larger than VRAM, or a model on CPU only | llama.cpp endpoint |
| Serve any model behind your own REST API | FastAPI endpoint |
| Serve a model with Ray Serve, LitServe, BentoML, or another framework | Other frameworks |
vLLMDeploy a high-throughput vLLM model server.SGLangDeploy an SGLang model server for fast structured generation.OllamaServe open models locally with Ollama’s OpenAI-compatible API.llama.cppServe quantized GGUF models with llama-server as a Flyte app, on GPU or CPU, behind an OpenAI-compatible API.FastAPIServe any Python model behind a custom REST API on your own compute with FastAPI, for models and request shapes an LLM server doesn’t cover.Other frameworksServe models with Ray Serve, LitServe, BentoML, or any other ML serving framework by running its server in a generic AppEnvironment.