Providers

Tasks and apps can call a hosted model API using a key from your own provider account. Billing, rate limits, and data handling stay under your agreement with the provider. You store the key as a secret, and your code reads it from an environment variable at runtime.

Supported providers

These guides cover setup and a complete task for the most common providers:

Other OpenAI-compatible providers

These providers work with the OpenAI SDK by changing the base URL. Start from the Fireworks AI task, and change the base URL, the secret, the environment variable, and the model.

Two of these differ from the task:

• Azure OpenAI: Set model to the name of your deployment, not the name of the underlying model.
• Perplexity: Its Agent API uses OpenAI’s Responses format, so call client.responses.create() instead of client.chat.completions.create(). Support for its earlier Sonar Chat Completions API has ended.

Providers with their own SDK

These providers don’t document an OpenAI-compatible endpoint for production use. Add the SDK to the image and pass it the key from the environment. See the Anthropic guide for the pattern.

Store the API key as a secret

Run this command without --value and paste the key at the prompt, which keeps it out of your shell history:

flyte create secret fireworks-api-key --project <your-project> --domain development

Leave out --project and --domain to make the secret available across your organization. Prefer a project and domain, so that one team’s tasks can’t use another team’s key.

Check that the secret exists:

flyte get secret --project <your-project> --domain development

The output lists metadata such as the name and scope. It never includes the value.

Call the provider from a task

Declare the secret on the environment and read it from the environment variable. This task calls Fireworks AI through the OpenAI SDK:

import asyncio
import os

import flyte

env = flyte.TaskEnvironment(
    name="fireworks-inference",
    image=flyte.Image.from_debian_base().with_pip_packages("openai"),
    secrets=flyte.Secret(key="fireworks-api-key", as_env_var="FIREWORKS_API_KEY"),
)

@env.task
async def summarize(text: str) -> str:
    from openai import AsyncOpenAI

    client = AsyncOpenAI(
        base_url="https://api.fireworks.ai/inference/v1",
        api_key=os.environ["FIREWORKS_API_KEY"],
    )
    response = await client.chat.completions.create(
        model="accounts/fireworks/models/glm-5p3-flash",
        messages=[{"role": "user", "content": f"Summarize in two sentences:\n\n{text}"}],
    )
    return response.choices[0].message.content

@env.task
async def main(texts: list[str]) -> list[str]:
    return await asyncio.gather(*(summarize(text) for text in texts))

if __name__ == "__main__":
    flyte.init_from_config()
    run = flyte.run(main, texts=["First document...", "Second document..."])
    print(run.url)

A task can only read a secret whose scope covers the project and domain it runs in. Provider model IDs change often, so check the provider’s model catalog for current names.

Call the provider from an app

AppEnvironment takes the same secrets argument, and the app reads the same environment variable:

app_env = FastAPIAppEnvironment(
    name="summarizer-api",
    app=app,
    image=flyte.Image.from_debian_base().with_pip_packages("fastapi", "uvicorn", "openai"),
    secrets=flyte.Secret(key="fireworks-api-key", as_env_var="FIREWORKS_API_KEY"),
)

Use a different key for one call

Override the secret for a single invocation when the same task needs different credentials, for example per team:

    team_b = summarize.override(
        secrets=flyte.Secret(key="fireworks-api-key-team-b", as_env_var="FIREWORKS_API_KEY")
    )
    result = await team_b(text=text)

The override replaces the environment’s secrets for that call. On a reusable container it raises an error unless you also pass reusable="off". See Overriding secrets.

Keep keys safe

• Never return a key from a task. Task outputs are written in plaintext to your object store and appear in the UI and in downstream tasks.
• Keep keys out of code, task inputs, and flyte.run() arguments. Inputs are stored and displayed the same way as outputs.
• Rotate without downtime. Create a key at the provider, delete and re-create the secret with the new value, redeploy apps that use it, and then revoke the old key.

Troubleshooting

The provider returns 401: The key is invalid, revoked, or was stored with a typo. Test it outside Flyte with the provider’s REST API, then re-create the secret.

The task fails before it runs, with a secret not found error: The secret’s scope doesn’t cover the task’s project and domain. List secrets with the same --project and --domain the task uses.

The provider returns 429: You’ve reached a rate limit on your provider account. Check its limits and your usage in the provider’s console.