Ollama app

Ollama is a lightweight runtime for serving open large language models (LLMs) locally, with a built-in OpenAI-compatible API. It is a good fit for smaller models and quick local-style serving, and complements the higher-throughput vLLM and SGLang integrations.

Unlike vLLM and SGLang, Ollama has no dedicated *AppEnvironment plugin. Instead, you serve it with the generic AppEnvironment: an image that installs Ollama plus a small entrypoint that launches ollama serve and pulls the model on startup.

Installation

Ollama needs no Flyte plugin — it is installed into the app image. Start from the default Flyte base image and add the Ollama binary with a build command:

basic_ollama.py 
# Install Ollama on top of the default Flyte base image. `install.sh` drops the
# `ollama` binary into /usr/local/bin (no systemd is needed inside a container).
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
)

The install.sh script drops the ollama binary into /usr/local/bin; no systemd service is needed inside a container.

Basic Ollama app

Define the app with a GPU-backed AppEnvironment. The args run the script’s --server entrypoint, which starts Ollama and pulls the model:

basic_ollama.py 
ollama_app = flyte.app.AppEnvironment(
    name="ollama-app",
    image=image,
    args=["python", file_name, "--server"],
    port=PORT,
    resources=flyte.Resources(
        cpu="4",
        memory="16Gi",
        gpu="L40s:1",  # GPU accelerates inference; omit to run small models on CPU
        disk="20Gi",
    ),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),
        scaledown_after=300,  # Scale down after 5 minutes of inactivity
    ),
    requires_auth=False,
)

The --server entrypoint binds Ollama to the app port, waits for it to be ready, and pulls the model so the OpenAI-compatible endpoint can serve it:

basic_ollama.py 
def serve() -> None:
    """Start `ollama serve`, wait for it, pull the model, then block."""
    # Bind to all interfaces so the platform can route to the server.
    server = subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{PORT}"},
    )

    # Wait for the server to accept connections.
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)

    # Pull the model so the OpenAI-compatible endpoint can serve it.
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{PORT}"},
        check=True,
    )
    print(f"Ollama serving '{MODEL}' on port {PORT}")

    server.wait()

Deploy it with flyte.serve:

basic_ollama.py 
if __name__ == "__main__":
    if "--server" in sys.argv:
        serve()
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(ollama_app)
        print(f"Deployed Ollama app: {app.url}")

Choosing a model

MODEL can be any tag from the Ollama library — for example qwen3:0.6b, llama3.2:1b, or gemma3:1b. Larger models need more GPU memory and disk; size the resources accordingly.

Small models run comfortably on CPU. To run without a GPU, drop the gpu field from resources. A GPU is recommended for larger models or higher throughput.

Using the OpenAI-compatible API

Once deployed, the Ollama app exposes an OpenAI-compatible API. Call it exactly like the vLLM or SGLang apps:

from openai import OpenAI

client = OpenAI(
    base_url="https://your-app-url/v1",  # Ollama endpoint
    api_key="ollama",  # Ollama ignores the key; any non-empty string works
)

response = client.chat.completions.create(
    model="qwen3:0.6b",  # Your MODEL tag
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ],
)

print(response.choices[0].message.content)

Chat UI with Streamlit

To put a browser UI in front of the model, run Ollama internally and expose a Streamlit chat interface instead of the raw API. Only the Streamlit port is exposed; the browser talks to Streamlit, and Streamlit talks to Ollama over localhost:

ollama_streamlit.py 
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
    .with_pip_packages("streamlit==1.41.1", "openai")
)

app_env = flyte.app.AppEnvironment(
    name="ollama-streamlit",
    image=image,
    args=["python", file_name, "--server"],
    port=APP_PORT,
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L40s:1", disk="20Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 1), scaledown_after=300),
    requires_auth=False,
)

The UI is a standard Streamlit chat app that points its OpenAI client at the local Ollama server:

ollama_streamlit.py 
def render_ui() -> None:
    """The Streamlit chat UI. Talks to the local Ollama OpenAI-compatible API."""
    import streamlit as st
    from openai import OpenAI

    st.set_page_config(page_title="Ollama Chat", page_icon="🦙")
    st.title("🦙 Ollama Chat")

    client = OpenAI(base_url=f"http://127.0.0.1:{OLLAMA_PORT}/v1", api_key="ollama")

    if "messages" not in st.session_state:
        st.session_state.messages = []

    for msg in st.session_state.messages:
        st.chat_message(msg["role"]).write(msg["content"])

    if prompt := st.chat_input("Ask something..."):
        st.session_state.messages.append({"role": "user", "content": prompt})
        st.chat_message("user").write(prompt)
        response = client.chat.completions.create(
            model=MODEL, messages=st.session_state.messages
        )
        answer = response.choices[0].message.content
        st.session_state.messages.append({"role": "assistant", "content": answer})
        st.chat_message("assistant").write(answer)

A helper starts Ollama in the background and pulls the model:

ollama_streamlit.py 
def start_ollama() -> None:
    """Launch `ollama serve` in the background and pull the model."""
    subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{OLLAMA_PORT}"},
    )
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{OLLAMA_PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{OLLAMA_PORT}"},
        check=True,
    )

The container entrypoint runs that helper, then hands the exposed port to Streamlit:

ollama_streamlit.py 
if __name__ == "__main__":
    if "--ui" in sys.argv:
        # Re-entry: Streamlit runs this file to render the UI.
        render_ui()
    elif "--server" in sys.argv:
        # Container entrypoint: start Ollama, then hand the port to Streamlit.
        start_ollama()
        subprocess.run(
            [
                "streamlit", "run", file_name,
                "--server.port", str(APP_PORT),
                "--server.address", "0.0.0.0",
                "--", "--ui",
            ],
            check=True,
        )
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(app_env)
        print(f"Deployed app: {app.url}")

Authentication

The generic AppEnvironment uses Union’s platform-level authentication. Leave requires_auth=True (the default) to require an authenticated caller, or set requires_auth=False for a public endpoint. Ollama has no built-in API-key argument of its own, so prefer platform auth over exposing a public endpoint. For app-managed authentication (verifying a Bearer token yourself with a Flyte secret), see Secret-based authentication.

Calling an authenticated app

With requires_auth=True (the default), callers authenticate with a Union API key. Create one with the CLI:

flyte create api-key --name ollama-app-key

This prints an export FLYTE_API_KEY="..." command. Pass the key in the Authorization: Bearer header when you invoke the endpoint:

curl -X POST "https://your-app-url/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FLYTE_API_KEY" \
  -d '{
    "model": "qwen3:0.6b",
    "messages": [{"role": "user", "content": "Hello, how are you?"}]
  }'

When calling through the OpenAI client, pass the same key as api_key — it becomes the Authorization: Bearer token (Ollama itself ignores the value). See Using API keys with Union apps for details.

Autoscaling

Ollama apps work well with scale-to-zero, so an idle model server costs nothing:

app_env = flyte.app.AppEnvironment(
    name="autoscaling-ollama-app",
    image=image,
    args=["python", "basic_ollama.py", "--server"],
    port=8080,
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L40s:1", disk="20Gi"),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),  # Scale to zero when idle
        scaledown_after=600,  # 10 minutes idle before scaling down
    ),
    requires_auth=True,
)

Because the model is pulled at startup, a larger scaledown_after avoids re-pulling on every cold start.

Best practices

  1. Right-size resources: Match GPU memory and disk to the model. Small models run on CPU; drop the gpu field to save cost.
  2. Bake big models into the image: For faster, more reproducible cold starts, ollama pull the model in a build command instead of at startup.
  3. Use scale-to-zero: Set an appropriate scaledown_after to balance cost against cold-start latency.
  4. Prefer platform auth: Ollama has no native API-key auth, so rely on requires_auth rather than exposing a public endpoint.
  5. Pick the right runtime: Use Ollama for lightweight or local-style serving; reach for vLLM or SGLang for high-throughput production inference.

Troubleshooting

Model pull fails or times out:

  • Verify the MODEL tag exists in the Ollama library
  • Increase disk in resources for larger models
  • Review container logs for the ollama pull output

Server not reachable:

  • Confirm Ollama is bound to 0.0.0.0 on the app port via OLLAMA_HOST
  • Check that the app port matches the port Ollama serves on

Slow first response:

  • The model is pulled on startup; use a larger scaledown_after or bake the model into the image
  • Use a smaller model, or add a GPU for faster inference