llama.cpp

The flyteplugins-llamacpp plugin serves GGUF models with llama.cpp’s llama-server as a Flyte app. LlamaCppAppEnvironment exposes an OpenAI-compatible API under /v1 and the llama.cpp Web UI at /.

Use it for quantized GGUF weights, for models larger than your GPU’s VRAM, and for CPU-only serving. For high-throughput serving of full-precision weights, use vLLM or SGLang.

Installation

pip install flyteplugins-llamacpp

Quickstart

Set model_hf_path to a GGUF repo, with an optional quantization tag. llama-server downloads the weights when the app starts:

llamacpp_apps.py
                
gemma_app = LlamaCppAppEnvironment(
    name="gemma-app",
    model_hf_path="ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
    model_id="gemma-3-4b-it",
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L4:1", disk="50Gi"),
)

Serve the app:

llamacpp_apps.py
                
if __name__ == "__main__":
    flyte.init_from_config()
    app = flyte.serve(gemma_app)
    print(f"Deployed llama.cpp app: {app.url}")

Next steps

The llama.cpp endpoint guide covers object storage, calling the app, choosing an image, llama-server arguments, and speculative decoding.

API reference

See the llama.cpp API reference.