llama.cpp

The flyteplugins-llamacpp plugin serves GGUF models with llama.cpp’s llama-server as a Flyte app. LlamaCppAppEnvironment exposes an OpenAI-compatible API under /v1 and the llama.cpp Web UI at /.

Use it for quantized GGUF weights, for models larger than your GPU’s VRAM, and for CPU-only serving. For high-throughput serving of full-precision weights, use vLLM or SGLang.

Installation

pip install flyteplugins-llamacpp

Serve a model from Hugging Face

Set model_hf_path to a GGUF repo, with an optional quantization tag. llama-server downloads the weights when the app starts:

llamacpp_apps.py
                
gemma_app = LlamaCppAppEnvironment(
    name="gemma-app",
    model_hf_path="ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
    model_id="gemma-3-4b-it",
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L4:1", disk="50Gi"),
)

Serve the app:

llamacpp_apps.py
                
if __name__ == "__main__":
    flyte.init_from_config()
    app = flyte.serve(gemma_app)
    print(f"Deployed llama.cpp app: {app.url}")

Serve a model from object storage

Set model_path to a directory of GGUF weights, or to one .gguf file:

llamacpp_apps.py
                
llama_app = LlamaCppAppEnvironment(
    name="my-llm-app",
    # A directory of GGUF weights, or a direct path to one .gguf file.
    model_path="s3://<your-bucket>/models/<your-model-gguf>",
    model_id="my-model",
    resources=flyte.Resources(cpu="4", memory="32Gi", gpu="L40s:1", disk="100Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 1), scaledown_after=300),
)

The weights are downloaded into the container, so set disk to fit them. If the directory holds a sharded model, the app selects the -00001-of- shard and llama-server finds the rest. If it holds several unrelated .gguf files, the app serves the first one by sorted path.

model_path also accepts a flyte.app.RunOutput, to serve the output of a task. See Deploy trained models.

It accepts a flyte.app.ArtifactValue too, to serve a model artifact.

Set model_path or model_hf_path, not both. model_id is required, and it’s the name callers pass as model.

Call the app

Use any OpenAI client:

call_app.py
                
from openai import OpenAI

client = OpenAI(
    base_url="https://<your-app-url>/v1",
    api_key="<your-api-key>",  # A platform API key when the app has requires_auth=True
)

response = client.chat.completions.create(
    model="gemma-3-4b-it",  # The app's model_id
    messages=[{"role": "user", "content": "Hello, how are you?"}],
)

print(response.choices[0].message.content)

llama-server checks no key of its own unless you pass --api-key in extra_args.

With requires_auth=True (the default), pass a Union API key as api_key. See Calling an authenticated app.

Choose an image

The default image compiles llama-server from the tip of llama.cpp’s default branch, with CUDA for L4 and L40S GPUs (compute capability 8.9), so two builds can produce different servers. The source build makes the first image build slow.

To target other GPUs, or to pin a llama.cpp release for reproducible builds, use build_llama_cpp_image():

llamacpp_apps.py
                
fat_binary_app = LlamaCppAppEnvironment(
    name="fat-binary-app",
    model_hf_path="ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
    model_id="gemma-3-4b-it",
    image=build_llama_cpp_image(
        cuda_arch="80;86;89;90",  # A100, A10, L4 and L40S, H100
        ref="b6148",  # A llama.cpp release tag
    ),
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L4:1", disk="50Gi"),
)

To serve small models without a GPU, build a CPU-only image:

llamacpp_apps.py
                
cpu_app = LlamaCppAppEnvironment(
    name="cpu-app",
    model_hf_path="ggml-org/Qwen3-0.6B-GGUF:Q8_0",
    model_id="qwen3-0.6b",
    image=build_llama_cpp_image(cuda=False),
    resources=flyte.Resources(cpu="4", memory="8Gi", disk="10Gi"),
)

Match the image to the GPU

The default image targets only compute capability 8.9. On another GPU, such as an A100 or H100, set cuda_arch to include it.

Pass arguments to llama-server

extra_args is appended to the llama-server command, as a string or a list:

llamacpp_apps.py
                
tuned_app = LlamaCppAppEnvironment(
    name="tuned-app",
    model_hf_path="ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
    model_id="gemma-3-4b-it",
    extra_args="--ctx-size 32768 --parallel 4 --jinja",
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L4:1", disk="50Gi"),
)

Flag	Purpose
`--ctx-size`	Context length
`--parallel`	Concurrent request slots
`--jinja`	Use the model’s chat template, which tool calling needs
`--n-gpu-layers`	Limit GPU offload for a model larger than VRAM
`--cache-type-k`, `--cache-type-v`	Quantize the KV cache
`--flash-attn`	Enable flash attention

Values with spaces or JSON are quoted for you. An argument of the form $MY_VAR isn’t quoted, and Flyte expands it from the app’s environment.

See the llama-server docs for all flags.

Use speculative decoding

Set draft_model_path or draft_model_hf_path to a small draft GGUF, and tune it with extra_args:

llamacpp_apps.py
                
speculative_app = LlamaCppAppEnvironment(
    name="speculative-app",
    model_path="s3://<your-bucket>/models/qwen3-32b-gguf",
    model_id="qwen3-32b",
    draft_model_hf_path="ggml-org/Qwen3-0.6B-GGUF:Q8_0",
    extra_args="--draft-max 16 --draft-min 1 --gpu-layers-draft 99",
    resources=flyte.Resources(cpu="8", memory="64Gi", gpu="L40s:1", disk="120Gi"),
)

Limitations

LlamaCppAppEnvironment sets its own command and parameters. Passing args or parameters raises a ValueError, so use extra_args and model_path instead.

API reference

See the llama.cpp API reference.