# Ollama app

[Ollama](https://ollama.com/) is a lightweight runtime for serving open large language models (LLMs) locally, with a built-in OpenAI-compatible API. It is a good fit for smaller models and quick local-style serving, and complements the higher-throughput [vLLM](https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/ollama-app/vllm-app) and [SGLang](https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/ollama-app/sglang-app) integrations.

Unlike vLLM and SGLang, Ollama has no dedicated `*AppEnvironment` plugin. Instead, you serve it with the generic [`AppEnvironment`](https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/build-apps/single-script-apps): an image that installs Ollama plus a small entrypoint that launches `ollama serve` and pulls the model on startup.

## Installation

Ollama needs no Flyte plugin — it is installed into the app image. Start from the default Flyte base image and add the Ollama binary with a build command:

```
# /// script
# requires-python = ">=3.12"
# dependencies = [
#    "flyte>=2.0.0b52",
# ]
# ///

"""A simple Ollama serving app example.

Unlike vLLM and SGLang, Ollama has no dedicated ``*AppEnvironment`` plugin, so it
is served with the generic ``flyte.app.AppEnvironment``: an image that installs
Ollama plus a small ``--server`` entrypoint that launches ``ollama serve`` and
pulls the model on startup. Ollama exposes an OpenAI-compatible API, so clients
call it exactly like the vLLM / SGLang apps.
"""

import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import flyte
import flyte.app

# Any tag from https://ollama.com/library. Small models run fine on CPU; larger
# ones benefit from the GPU requested below.
MODEL = "qwen3:0.6b"

# Bind Ollama to the app port so the platform can route to it.
PORT = 8080

file_name = Path(__file__).name

# {{docs-fragment ollama-image}}
# Install Ollama on top of the default Flyte base image. `install.sh` drops the
# `ollama` binary into /usr/local/bin (no systemd is needed inside a container).
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
)
# {{/docs-fragment ollama-image}}

# {{docs-fragment ollama-app}}
ollama_app = flyte.app.AppEnvironment(
    name="ollama-app",
    image=image,
    args=["python", file_name, "--server"],
    port=PORT,
    resources=flyte.Resources(
        cpu="4",
        memory="16Gi",
        gpu="L40s:1",  # GPU accelerates inference; omit to run small models on CPU
        disk="20Gi",
    ),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),
        scaledown_after=300,  # Scale down after 5 minutes of inactivity
    ),
    requires_auth=False,
)
# {{/docs-fragment ollama-app}}

# {{docs-fragment server}}
def serve() -> None:
    """Start `ollama serve`, wait for it, pull the model, then block."""
    # Bind to all interfaces so the platform can route to the server.
    server = subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{PORT}"},
    )

    # Wait for the server to accept connections.
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)

    # Pull the model so the OpenAI-compatible endpoint can serve it.
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{PORT}"},
        check=True,
    )
    print(f"Ollama serving '{MODEL}' on port {PORT}")

    server.wait()
# {{/docs-fragment server}}

# {{docs-fragment deploy}}
if __name__ == "__main__":
    if "--server" in sys.argv:
        serve()
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(ollama_app)
        print(f"Deployed Ollama app: {app.url}")
# {{/docs-fragment deploy}}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/ollama/basic_ollama.py*

The `install.sh` script drops the `ollama` binary into `/usr/local/bin`; no systemd service is needed inside a container.

## Basic Ollama app

Define the app with a GPU-backed `AppEnvironment`. The `args` run the script's `--server` entrypoint, which starts Ollama and pulls the model:

```
# /// script
# requires-python = ">=3.12"
# dependencies = [
#    "flyte>=2.0.0b52",
# ]
# ///

"""A simple Ollama serving app example.

Unlike vLLM and SGLang, Ollama has no dedicated ``*AppEnvironment`` plugin, so it
is served with the generic ``flyte.app.AppEnvironment``: an image that installs
Ollama plus a small ``--server`` entrypoint that launches ``ollama serve`` and
pulls the model on startup. Ollama exposes an OpenAI-compatible API, so clients
call it exactly like the vLLM / SGLang apps.
"""

import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import flyte
import flyte.app

# Any tag from https://ollama.com/library. Small models run fine on CPU; larger
# ones benefit from the GPU requested below.
MODEL = "qwen3:0.6b"

# Bind Ollama to the app port so the platform can route to it.
PORT = 8080

file_name = Path(__file__).name

# {{docs-fragment ollama-image}}
# Install Ollama on top of the default Flyte base image. `install.sh` drops the
# `ollama` binary into /usr/local/bin (no systemd is needed inside a container).
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
)
# {{/docs-fragment ollama-image}}

# {{docs-fragment ollama-app}}
ollama_app = flyte.app.AppEnvironment(
    name="ollama-app",
    image=image,
    args=["python", file_name, "--server"],
    port=PORT,
    resources=flyte.Resources(
        cpu="4",
        memory="16Gi",
        gpu="L40s:1",  # GPU accelerates inference; omit to run small models on CPU
        disk="20Gi",
    ),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),
        scaledown_after=300,  # Scale down after 5 minutes of inactivity
    ),
    requires_auth=False,
)
# {{/docs-fragment ollama-app}}

# {{docs-fragment server}}
def serve() -> None:
    """Start `ollama serve`, wait for it, pull the model, then block."""
    # Bind to all interfaces so the platform can route to the server.
    server = subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{PORT}"},
    )

    # Wait for the server to accept connections.
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)

    # Pull the model so the OpenAI-compatible endpoint can serve it.
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{PORT}"},
        check=True,
    )
    print(f"Ollama serving '{MODEL}' on port {PORT}")

    server.wait()
# {{/docs-fragment server}}

# {{docs-fragment deploy}}
if __name__ == "__main__":
    if "--server" in sys.argv:
        serve()
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(ollama_app)
        print(f"Deployed Ollama app: {app.url}")
# {{/docs-fragment deploy}}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/ollama/basic_ollama.py*

The `--server` entrypoint binds Ollama to the app port, waits for it to be ready, and pulls the model so the OpenAI-compatible endpoint can serve it:

```
# /// script
# requires-python = ">=3.12"
# dependencies = [
#    "flyte>=2.0.0b52",
# ]
# ///

"""A simple Ollama serving app example.

Unlike vLLM and SGLang, Ollama has no dedicated ``*AppEnvironment`` plugin, so it
is served with the generic ``flyte.app.AppEnvironment``: an image that installs
Ollama plus a small ``--server`` entrypoint that launches ``ollama serve`` and
pulls the model on startup. Ollama exposes an OpenAI-compatible API, so clients
call it exactly like the vLLM / SGLang apps.
"""

import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import flyte
import flyte.app

# Any tag from https://ollama.com/library. Small models run fine on CPU; larger
# ones benefit from the GPU requested below.
MODEL = "qwen3:0.6b"

# Bind Ollama to the app port so the platform can route to it.
PORT = 8080

file_name = Path(__file__).name

# {{docs-fragment ollama-image}}
# Install Ollama on top of the default Flyte base image. `install.sh` drops the
# `ollama` binary into /usr/local/bin (no systemd is needed inside a container).
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
)
# {{/docs-fragment ollama-image}}

# {{docs-fragment ollama-app}}
ollama_app = flyte.app.AppEnvironment(
    name="ollama-app",
    image=image,
    args=["python", file_name, "--server"],
    port=PORT,
    resources=flyte.Resources(
        cpu="4",
        memory="16Gi",
        gpu="L40s:1",  # GPU accelerates inference; omit to run small models on CPU
        disk="20Gi",
    ),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),
        scaledown_after=300,  # Scale down after 5 minutes of inactivity
    ),
    requires_auth=False,
)
# {{/docs-fragment ollama-app}}

# {{docs-fragment server}}
def serve() -> None:
    """Start `ollama serve`, wait for it, pull the model, then block."""
    # Bind to all interfaces so the platform can route to the server.
    server = subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{PORT}"},
    )

    # Wait for the server to accept connections.
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)

    # Pull the model so the OpenAI-compatible endpoint can serve it.
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{PORT}"},
        check=True,
    )
    print(f"Ollama serving '{MODEL}' on port {PORT}")

    server.wait()
# {{/docs-fragment server}}

# {{docs-fragment deploy}}
if __name__ == "__main__":
    if "--server" in sys.argv:
        serve()
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(ollama_app)
        print(f"Deployed Ollama app: {app.url}")
# {{/docs-fragment deploy}}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/ollama/basic_ollama.py*

Deploy it with `flyte.serve`:

```
# /// script
# requires-python = ">=3.12"
# dependencies = [
#    "flyte>=2.0.0b52",
# ]
# ///

"""A simple Ollama serving app example.

Unlike vLLM and SGLang, Ollama has no dedicated ``*AppEnvironment`` plugin, so it
is served with the generic ``flyte.app.AppEnvironment``: an image that installs
Ollama plus a small ``--server`` entrypoint that launches ``ollama serve`` and
pulls the model on startup. Ollama exposes an OpenAI-compatible API, so clients
call it exactly like the vLLM / SGLang apps.
"""

import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import flyte
import flyte.app

# Any tag from https://ollama.com/library. Small models run fine on CPU; larger
# ones benefit from the GPU requested below.
MODEL = "qwen3:0.6b"

# Bind Ollama to the app port so the platform can route to it.
PORT = 8080

file_name = Path(__file__).name

# {{docs-fragment ollama-image}}
# Install Ollama on top of the default Flyte base image. `install.sh` drops the
# `ollama` binary into /usr/local/bin (no systemd is needed inside a container).
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
)
# {{/docs-fragment ollama-image}}

# {{docs-fragment ollama-app}}
ollama_app = flyte.app.AppEnvironment(
    name="ollama-app",
    image=image,
    args=["python", file_name, "--server"],
    port=PORT,
    resources=flyte.Resources(
        cpu="4",
        memory="16Gi",
        gpu="L40s:1",  # GPU accelerates inference; omit to run small models on CPU
        disk="20Gi",
    ),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),
        scaledown_after=300,  # Scale down after 5 minutes of inactivity
    ),
    requires_auth=False,
)
# {{/docs-fragment ollama-app}}

# {{docs-fragment server}}
def serve() -> None:
    """Start `ollama serve`, wait for it, pull the model, then block."""
    # Bind to all interfaces so the platform can route to the server.
    server = subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{PORT}"},
    )

    # Wait for the server to accept connections.
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)

    # Pull the model so the OpenAI-compatible endpoint can serve it.
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{PORT}"},
        check=True,
    )
    print(f"Ollama serving '{MODEL}' on port {PORT}")

    server.wait()
# {{/docs-fragment server}}

# {{docs-fragment deploy}}
if __name__ == "__main__":
    if "--server" in sys.argv:
        serve()
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(ollama_app)
        print(f"Deployed Ollama app: {app.url}")
# {{/docs-fragment deploy}}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/ollama/basic_ollama.py*

## Choosing a model

`MODEL` can be any tag from the [Ollama library](https://ollama.com/library) — for example `qwen3:0.6b`, `llama3.2:1b`, or `gemma3:1b`. Larger models need more GPU memory and disk; size the `resources` accordingly.

> [!NOTE]
> Small models run comfortably on CPU. To run without a GPU, drop the `gpu` field from `resources`. A GPU is recommended for larger models or higher throughput.

## Using the OpenAI-compatible API

Once deployed, the Ollama app exposes an OpenAI-compatible API. Call it exactly like the vLLM or SGLang apps:

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://your-app-url/v1",  # Ollama endpoint
    api_key="ollama",  # Ollama ignores the key; any non-empty string works
)

response = client.chat.completions.create(
    model="qwen3:0.6b",  # Your MODEL tag
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ],
)

print(response.choices[0].message.content)
```

## Chat UI with Streamlit

To put a browser UI in front of the model, run Ollama internally and expose a [Streamlit](https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/ollama-app/streamlit-app) chat interface instead of the raw API. Only the Streamlit port is exposed; the browser talks to Streamlit, and Streamlit talks to Ollama over localhost:

```
# /// script
# requires-python = ">=3.12"
# dependencies = [
#    "flyte>=2.0.0b52",
#    "streamlit",
#    "openai",
# ]
# ///

"""An Ollama chat app fronted by a Streamlit UI.

Ollama runs internally as an OpenAI-compatible server; a Streamlit chat
interface fronts it. Only the Streamlit port is exposed to the platform — the
browser talks to Streamlit, and Streamlit talks to Ollama over localhost.
"""

import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import flyte
import flyte.app

MODEL = "qwen3:0.6b"
OLLAMA_PORT = 11434  # internal only, not exposed
APP_PORT = 8080  # Streamlit UI, exposed to the platform

file_name = Path(__file__).name

# {{docs-fragment app-env}}
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
    .with_pip_packages("streamlit==1.41.1", "openai")
)

app_env = flyte.app.AppEnvironment(
    name="ollama-streamlit",
    image=image,
    args=["python", file_name, "--server"],
    port=APP_PORT,
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L40s:1", disk="20Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 1), scaledown_after=300),
    requires_auth=False,
)
# {{/docs-fragment app-env}}

# {{docs-fragment ui}}
def render_ui() -> None:
    """The Streamlit chat UI. Talks to the local Ollama OpenAI-compatible API."""
    import streamlit as st
    from openai import OpenAI

    st.set_page_config(page_title="Ollama Chat", page_icon="🦙")
    st.title("🦙 Ollama Chat")

    client = OpenAI(base_url=f"http://127.0.0.1:{OLLAMA_PORT}/v1", api_key="ollama")

    if "messages" not in st.session_state:
        st.session_state.messages = []

    for msg in st.session_state.messages:
        st.chat_message(msg["role"]).write(msg["content"])

    if prompt := st.chat_input("Ask something..."):
        st.session_state.messages.append({"role": "user", "content": prompt})
        st.chat_message("user").write(prompt)
        response = client.chat.completions.create(
            model=MODEL, messages=st.session_state.messages
        )
        answer = response.choices[0].message.content
        st.session_state.messages.append({"role": "assistant", "content": answer})
        st.chat_message("assistant").write(answer)
# {{/docs-fragment ui}}

# {{docs-fragment server}}
def start_ollama() -> None:
    """Launch `ollama serve` in the background and pull the model."""
    subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{OLLAMA_PORT}"},
    )
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{OLLAMA_PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{OLLAMA_PORT}"},
        check=True,
    )
# {{/docs-fragment server}}

# {{docs-fragment deploy}}
if __name__ == "__main__":
    if "--ui" in sys.argv:
        # Re-entry: Streamlit runs this file to render the UI.
        render_ui()
    elif "--server" in sys.argv:
        # Container entrypoint: start Ollama, then hand the port to Streamlit.
        start_ollama()
        subprocess.run(
            [
                "streamlit", "run", file_name,
                "--server.port", str(APP_PORT),
                "--server.address", "0.0.0.0",
                "--", "--ui",
            ],
            check=True,
        )
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(app_env)
        print(f"Deployed app: {app.url}")
# {{/docs-fragment deploy}}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/ollama/ollama_streamlit.py*

The UI is a standard Streamlit chat app that points its OpenAI client at the local Ollama server:

```
# /// script
# requires-python = ">=3.12"
# dependencies = [
#    "flyte>=2.0.0b52",
#    "streamlit",
#    "openai",
# ]
# ///

"""An Ollama chat app fronted by a Streamlit UI.

Ollama runs internally as an OpenAI-compatible server; a Streamlit chat
interface fronts it. Only the Streamlit port is exposed to the platform — the
browser talks to Streamlit, and Streamlit talks to Ollama over localhost.
"""

import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import flyte
import flyte.app

MODEL = "qwen3:0.6b"
OLLAMA_PORT = 11434  # internal only, not exposed
APP_PORT = 8080  # Streamlit UI, exposed to the platform

file_name = Path(__file__).name

# {{docs-fragment app-env}}
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
    .with_pip_packages("streamlit==1.41.1", "openai")
)

app_env = flyte.app.AppEnvironment(
    name="ollama-streamlit",
    image=image,
    args=["python", file_name, "--server"],
    port=APP_PORT,
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L40s:1", disk="20Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 1), scaledown_after=300),
    requires_auth=False,
)
# {{/docs-fragment app-env}}

# {{docs-fragment ui}}
def render_ui() -> None:
    """The Streamlit chat UI. Talks to the local Ollama OpenAI-compatible API."""
    import streamlit as st
    from openai import OpenAI

    st.set_page_config(page_title="Ollama Chat", page_icon="🦙")
    st.title("🦙 Ollama Chat")

    client = OpenAI(base_url=f"http://127.0.0.1:{OLLAMA_PORT}/v1", api_key="ollama")

    if "messages" not in st.session_state:
        st.session_state.messages = []

    for msg in st.session_state.messages:
        st.chat_message(msg["role"]).write(msg["content"])

    if prompt := st.chat_input("Ask something..."):
        st.session_state.messages.append({"role": "user", "content": prompt})
        st.chat_message("user").write(prompt)
        response = client.chat.completions.create(
            model=MODEL, messages=st.session_state.messages
        )
        answer = response.choices[0].message.content
        st.session_state.messages.append({"role": "assistant", "content": answer})
        st.chat_message("assistant").write(answer)
# {{/docs-fragment ui}}

# {{docs-fragment server}}
def start_ollama() -> None:
    """Launch `ollama serve` in the background and pull the model."""
    subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{OLLAMA_PORT}"},
    )
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{OLLAMA_PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{OLLAMA_PORT}"},
        check=True,
    )
# {{/docs-fragment server}}

# {{docs-fragment deploy}}
if __name__ == "__main__":
    if "--ui" in sys.argv:
        # Re-entry: Streamlit runs this file to render the UI.
        render_ui()
    elif "--server" in sys.argv:
        # Container entrypoint: start Ollama, then hand the port to Streamlit.
        start_ollama()
        subprocess.run(
            [
                "streamlit", "run", file_name,
                "--server.port", str(APP_PORT),
                "--server.address", "0.0.0.0",
                "--", "--ui",
            ],
            check=True,
        )
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(app_env)
        print(f"Deployed app: {app.url}")
# {{/docs-fragment deploy}}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/ollama/ollama_streamlit.py*

A helper starts Ollama in the background and pulls the model:

```
# /// script
# requires-python = ">=3.12"
# dependencies = [
#    "flyte>=2.0.0b52",
#    "streamlit",
#    "openai",
# ]
# ///

"""An Ollama chat app fronted by a Streamlit UI.

Ollama runs internally as an OpenAI-compatible server; a Streamlit chat
interface fronts it. Only the Streamlit port is exposed to the platform — the
browser talks to Streamlit, and Streamlit talks to Ollama over localhost.
"""

import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import flyte
import flyte.app

MODEL = "qwen3:0.6b"
OLLAMA_PORT = 11434  # internal only, not exposed
APP_PORT = 8080  # Streamlit UI, exposed to the platform

file_name = Path(__file__).name

# {{docs-fragment app-env}}
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
    .with_pip_packages("streamlit==1.41.1", "openai")
)

app_env = flyte.app.AppEnvironment(
    name="ollama-streamlit",
    image=image,
    args=["python", file_name, "--server"],
    port=APP_PORT,
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L40s:1", disk="20Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 1), scaledown_after=300),
    requires_auth=False,
)
# {{/docs-fragment app-env}}

# {{docs-fragment ui}}
def render_ui() -> None:
    """The Streamlit chat UI. Talks to the local Ollama OpenAI-compatible API."""
    import streamlit as st
    from openai import OpenAI

    st.set_page_config(page_title="Ollama Chat", page_icon="🦙")
    st.title("🦙 Ollama Chat")

    client = OpenAI(base_url=f"http://127.0.0.1:{OLLAMA_PORT}/v1", api_key="ollama")

    if "messages" not in st.session_state:
        st.session_state.messages = []

    for msg in st.session_state.messages:
        st.chat_message(msg["role"]).write(msg["content"])

    if prompt := st.chat_input("Ask something..."):
        st.session_state.messages.append({"role": "user", "content": prompt})
        st.chat_message("user").write(prompt)
        response = client.chat.completions.create(
            model=MODEL, messages=st.session_state.messages
        )
        answer = response.choices[0].message.content
        st.session_state.messages.append({"role": "assistant", "content": answer})
        st.chat_message("assistant").write(answer)
# {{/docs-fragment ui}}

# {{docs-fragment server}}
def start_ollama() -> None:
    """Launch `ollama serve` in the background and pull the model."""
    subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{OLLAMA_PORT}"},
    )
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{OLLAMA_PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{OLLAMA_PORT}"},
        check=True,
    )
# {{/docs-fragment server}}

# {{docs-fragment deploy}}
if __name__ == "__main__":
    if "--ui" in sys.argv:
        # Re-entry: Streamlit runs this file to render the UI.
        render_ui()
    elif "--server" in sys.argv:
        # Container entrypoint: start Ollama, then hand the port to Streamlit.
        start_ollama()
        subprocess.run(
            [
                "streamlit", "run", file_name,
                "--server.port", str(APP_PORT),
                "--server.address", "0.0.0.0",
                "--", "--ui",
            ],
            check=True,
        )
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(app_env)
        print(f"Deployed app: {app.url}")
# {{/docs-fragment deploy}}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/ollama/ollama_streamlit.py*

The container entrypoint runs that helper, then hands the exposed port to Streamlit:

```
# /// script
# requires-python = ">=3.12"
# dependencies = [
#    "flyte>=2.0.0b52",
#    "streamlit",
#    "openai",
# ]
# ///

"""An Ollama chat app fronted by a Streamlit UI.

Ollama runs internally as an OpenAI-compatible server; a Streamlit chat
interface fronts it. Only the Streamlit port is exposed to the platform — the
browser talks to Streamlit, and Streamlit talks to Ollama over localhost.
"""

import os
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import flyte
import flyte.app

MODEL = "qwen3:0.6b"
OLLAMA_PORT = 11434  # internal only, not exposed
APP_PORT = 8080  # Streamlit UI, exposed to the platform

file_name = Path(__file__).name

# {{docs-fragment app-env}}
image = (
    flyte.Image.from_debian_base(python_version=(3, 12))
    .with_apt_packages("curl")
    .with_commands(["curl -fsSL https://ollama.com/install.sh | sh"])
    .with_pip_packages("streamlit==1.41.1", "openai")
)

app_env = flyte.app.AppEnvironment(
    name="ollama-streamlit",
    image=image,
    args=["python", file_name, "--server"],
    port=APP_PORT,
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L40s:1", disk="20Gi"),
    scaling=flyte.app.Scaling(replicas=(0, 1), scaledown_after=300),
    requires_auth=False,
)
# {{/docs-fragment app-env}}

# {{docs-fragment ui}}
def render_ui() -> None:
    """The Streamlit chat UI. Talks to the local Ollama OpenAI-compatible API."""
    import streamlit as st
    from openai import OpenAI

    st.set_page_config(page_title="Ollama Chat", page_icon="🦙")
    st.title("🦙 Ollama Chat")

    client = OpenAI(base_url=f"http://127.0.0.1:{OLLAMA_PORT}/v1", api_key="ollama")

    if "messages" not in st.session_state:
        st.session_state.messages = []

    for msg in st.session_state.messages:
        st.chat_message(msg["role"]).write(msg["content"])

    if prompt := st.chat_input("Ask something..."):
        st.session_state.messages.append({"role": "user", "content": prompt})
        st.chat_message("user").write(prompt)
        response = client.chat.completions.create(
            model=MODEL, messages=st.session_state.messages
        )
        answer = response.choices[0].message.content
        st.session_state.messages.append({"role": "assistant", "content": answer})
        st.chat_message("assistant").write(answer)
# {{/docs-fragment ui}}

# {{docs-fragment server}}
def start_ollama() -> None:
    """Launch `ollama serve` in the background and pull the model."""
    subprocess.Popen(
        ["ollama", "serve"],
        env={**os.environ, "OLLAMA_HOST": f"0.0.0.0:{OLLAMA_PORT}"},
    )
    for _ in range(60):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{OLLAMA_PORT}/api/version", timeout=2)
            break
        except Exception:
            time.sleep(1)
    subprocess.run(
        ["ollama", "pull", MODEL],
        env={**os.environ, "OLLAMA_HOST": f"127.0.0.1:{OLLAMA_PORT}"},
        check=True,
    )
# {{/docs-fragment server}}

# {{docs-fragment deploy}}
if __name__ == "__main__":
    if "--ui" in sys.argv:
        # Re-entry: Streamlit runs this file to render the UI.
        render_ui()
    elif "--server" in sys.argv:
        # Container entrypoint: start Ollama, then hand the port to Streamlit.
        start_ollama()
        subprocess.run(
            [
                "streamlit", "run", file_name,
                "--server.port", str(APP_PORT),
                "--server.address", "0.0.0.0",
                "--", "--ui",
            ],
            check=True,
        )
    else:
        flyte.init_from_config(root_dir=Path(__file__).parent)
        app = flyte.serve(app_env)
        print(f"Deployed app: {app.url}")
# {{/docs-fragment deploy}}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/user-guide/build-apps/ollama/ollama_streamlit.py*

## Authentication

The generic `AppEnvironment` uses Union's platform-level authentication. Leave `requires_auth=True` (the default) to require an authenticated caller, or set `requires_auth=False` for a public endpoint. Ollama has no built-in API-key argument of its own, so prefer platform auth over exposing a public endpoint. For app-managed authentication (verifying a Bearer token yourself with a Flyte secret), see [Secret-based authentication](https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/build-apps/secret-based-authentication).

### Calling an authenticated app

With `requires_auth=True` (the default), callers authenticate with a Union API key. Create one with the CLI:

```bash
flyte create api-key --name ollama-app-key
```

This prints an `export FLYTE_API_KEY="..."` command. Pass the key in the `Authorization: Bearer` header when you invoke the endpoint:

```bash
curl -X POST "https://your-app-url/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FLYTE_API_KEY" \
  -d '{
    "model": "qwen3:0.6b",
    "messages": [{"role": "user", "content": "Hello, how are you?"}]
  }'
```

When calling through the OpenAI client, pass the same key as `api_key` — it becomes the `Authorization: Bearer` token (Ollama itself ignores the value). See [Using API keys with Union apps](https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/authenticating#using-api-keys-with-union-apps) for details.

## Autoscaling

Ollama apps work well with scale-to-zero, so an idle model server costs nothing:

```python
app_env = flyte.app.AppEnvironment(
    name="autoscaling-ollama-app",
    image=image,
    args=["python", "basic_ollama.py", "--server"],
    port=8080,
    resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L40s:1", disk="20Gi"),
    scaling=flyte.app.Scaling(
        replicas=(0, 1),  # Scale to zero when idle
        scaledown_after=600,  # 10 minutes idle before scaling down
    ),
    requires_auth=True,
)
```

Because the model is pulled at startup, a larger `scaledown_after` avoids re-pulling on every cold start.

## Best practices

1. **Right-size resources**: Match GPU memory and disk to the model. Small models run on CPU; drop the `gpu` field to save cost.
2. **Bake big models into the image**: For faster, more reproducible cold starts, `ollama pull` the model in a build command instead of at startup.
3. **Use scale-to-zero**: Set an appropriate `scaledown_after` to balance cost against cold-start latency.
4. **Prefer platform auth**: Ollama has no native API-key auth, so rely on `requires_auth` rather than exposing a public endpoint.
5. **Pick the right runtime**: Use Ollama for lightweight or local-style serving; reach for [vLLM](https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/ollama-app/vllm-app) or [SGLang](https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/ollama-app/sglang-app) for high-throughput production inference.

## Troubleshooting

**Model pull fails or times out:**

- Verify the `MODEL` tag exists in the [Ollama library](https://ollama.com/library)
- Increase `disk` in `resources` for larger models
- Review container logs for the `ollama pull` output

**Server not reachable:**

- Confirm Ollama is bound to `0.0.0.0` on the app port via `OLLAMA_HOST`
- Check that the app `port` matches the port Ollama serves on

**Slow first response:**

- The model is pulled on startup; use a larger `scaledown_after` or bake the model into the image
- Use a smaller model, or add a GPU for faster inference

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/user-guide/native-app-integrations/ollama-app.md
**HTML**: https://www.union.ai/docs/v2/union/user-guide/native-app-integrations/ollama-app/