Use from tasks and apps

Tasks and apps on any of your Union clusters call the gateway like any OpenAI-compatible API: set the base URL to the gateway’s endpoint and pass a virtual key as the API key. A summarization task, a RAG app, and an agent all share one endpoint, and each gets its own budget through its own key.

Prerequisites

A deployed gateway and its endpoint, from Settings → Inference Gateway. See Deploy the gateway.
A virtual key for the task or app, stored as a Union secret:
```
flyte create secret llm-gateway-key sk-un-...
```

Call the gateway from a task

Mount the secret on the environment:

call_from_task.py
                
# The gateway's endpoint, from Settings > LLM Gateway in the Union console.
GATEWAY_URL = "https://<your-gateway-endpoint>"

env = flyte.TaskEnvironment(
    name="llm-gateway-task",
    image=flyte.Image.from_debian_base().with_pip_packages("openai"),
    # Store the virtual key once with: flyte create secret llm-gateway-key sk-un-...
    secrets=flyte.Secret(key="llm-gateway-key", as_env_var="LLM_GATEWAY_KEY"),
)

Then point the client at the gateway. Address models as provider/model:

call_from_task.py
                
@env.task
async def summarize(text: str) -> str:
    from openai import AsyncOpenAI

    client = AsyncOpenAI(base_url=f"{GATEWAY_URL}/v1", api_key=os.environ["LLM_GATEWAY_KEY"])
    response = await client.chat.completions.create(
        # Models are addressed as provider/model.
        model="openai/gpt-4.1-mini",
        messages=[{"role": "user", "content": f"Summarize in one sentence:\n\n{text}"}],
        # Tried in order if the primary model fails or is rejected for this key.
        extra_body={"fallbacks": ["anthropic/claude-haiku-4-5"]},
    )
    return response.choices[0].message.content

The fallbacks list is optional. If openai/gpt-4.1-mini fails, the gateway retries the call on anthropic/claude-haiku-4-5. See Fallbacks.

Call the gateway from an app

Apps use the same pattern. This FastAPI app has its own key, so its budget and allowed models are separate from every task’s. The x-un-dim-app header tags its requests for spend attribution:

call_from_app.py
                
                    
                
            
                
            

                
                    
                
            
app = FastAPI()

@app.get("/ask")
async def ask(question: str) -> dict[str, str]:
    from openai import AsyncOpenAI

    client = AsyncOpenAI(
        base_url=f"{GATEWAY_URL}/v1",
        api_key=os.environ["LLM_GATEWAY_KEY"],
        # Tag requests so the console can break down spend by app.
        default_headers={"x-un-dim-app": "support-bot"},
    )
    response = await client.chat.completions.create(
        model="anthropic/claude-sonnet-4-5",
        messages=[{"role": "user", "content": question}],
    )
    return {"answer": response.choices[0].message.content}

env = FastAPIAppEnvironment(
    name="support-bot",
    app=app,
    image=flyte.Image.from_debian_base().with_pip_packages("fastapi", "uvicorn", "openai"),
    # Each app gets its own virtual key, so its budget and allowed models are its own.
    secrets=flyte.Secret(key="support-bot-llm-key", as_env_var="LLM_GATEWAY_KEY"),
)

Use other clients

Any client that speaks the OpenAI API works against the gateway’s /v1 base URL:

Anthropic models through the OpenAI SDK. Use model="anthropic/<model>". The gateway translates the request.
LiteLLM. Use the openai/ prefix so LiteLLM sends an OpenAI-format request, set api_base to the gateway’s /v1 URL, and pass the virtual key as api_key.
LangChain. Use ChatOpenAI(base_url=..., api_key=..., model="provider/model").

curl.

        
    
curl https://<your-gateway-endpoint>/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-4.1-mini", "messages": [{"role": "user", "content": "Hello"}]}'

List the models a key can reach with GET /v1/models.