Run models from Flyte by calling a hosted provider, serving open-weight models on your own GPUs, or deploying the models your tasks train.
Inference
Choose how to run a model based on where it lives and how you call it:
| You want to | Use |
|---|---|
| Call a hosted model such as Fireworks, OpenAI, or Anthropic from a task or app | Providers |
| Serve an open-weight LLM on your own GPUs at high throughput | vLLM endpoint or SGLang endpoint |
| Serve a small LLM with minimal setup | Ollama endpoint |
| Serve quantized GGUF weights, a model larger than VRAM, or a model on CPU only | llama.cpp endpoint |
| Serve any model behind your own API | FastAPI endpoint or Other frameworks such as Ray Serve, LitServe, and BentoML |
| Copy Hugging Face weights into your storage before serving them | Prefetching models |
| Serve a model you trained in a task | Deploy trained models |
| Score a dataset on GPUs as a job, with no endpoint | Batch inference |
| Load weights inside a task from object storage | Stream model weights |
Most hosted providers, and vLLM, SGLang, Ollama, and llama.cpp endpoints, expose the OpenAI API or an API compatible with it. Code written against the OpenAI SDK moves between them by changing the base URL, the API key, and the model name. Some providers, such as Anthropic, need their own SDK. See Providers.
Route calls through the LLM Gateway
The LLM Gateway puts one endpoint in front of hosted providers, including Fireworks, and the models you serve with vLLM or SGLang. It adds budgets, fallbacks between providers, and guardrails. See Choose direct calls or the gateway.
ProvidersCall hosted LLM and model APIs such as Fireworks AI, OpenAI, and Anthropic from Flyte tasks and apps, with API keys stored as secrets.Self-hosted endpointsServe models on your own GPUs or CPUs, with vLLM, SGLang, Ollama, llama.cpp, FastAPI, Ray Serve, LitServe, BentoML, or another framework.Prefetching modelsDownload and shard a HuggingFace model before an inference app starts.Deploy trained modelsServe a model trained in a Flyte task behind an endpoint, with vLLM, SGLang, or your own FastAPI app.Batch inferenceKeep expensive GPUs busy during batch inference instead of idle and waiting.Stream model weightsLoad a model’s safetensors weights from object storage onto the GPU inside a task, without writing them to local disk.