Fireworks
Fireworks AI has an OpenAI-compatible API, so a Flyte task or app can call it with the OpenAI SDK by changing the base URL.
| Setting | Value |
|---|---|
| Base URL | https://api.fireworks.ai/inference/v1 |
| API key | API keys |
| Environment variable | FIREWORKS_API_KEY |
Store your Fireworks API key
Run this command and paste the key at the prompt:
flyte create secret fireworks-api-key --project <your-project> --domain developmentCall Fireworks from a task
import asyncio
import os
import flyte
env = flyte.TaskEnvironment(
name="fireworks-inference",
image=flyte.Image.from_debian_base().with_pip_packages("openai"),
secrets=flyte.Secret(key="fireworks-api-key", as_env_var="FIREWORKS_API_KEY"),
)
@env.task
async def summarize(text: str) -> str:
from openai import AsyncOpenAI
client = AsyncOpenAI(
base_url="https://api.fireworks.ai/inference/v1",
api_key=os.environ["FIREWORKS_API_KEY"],
)
response = await client.chat.completions.create(
model="accounts/fireworks/models/glm-5p3-flash",
messages=[{"role": "user", "content": f"Summarize in two sentences:\n\n{text}"}],
)
return response.choices[0].message.content
@env.task
async def main(texts: list[str]) -> list[str]:
return await asyncio.gather(*(summarize(text) for text in texts))
if __name__ == "__main__":
flyte.init_from_config()
run = flyte.run(main, texts=["First document...", "Second document..."])
print(run.url)
Models on Fireworks’ serverless tier use the name accounts/fireworks/models/<model>. Some catalog models aren’t available on that tier and need a dedicated deployment.
Call Fireworks from an app
An AppEnvironment takes the same secrets argument, and the app reads the same environment variable. See Call the provider from an app.
Use an on-demand deployment
An on-demand deployment runs a model on dedicated GPUs. To use one, change only the model name. The base URL, the key, and the secret stay the same:
# An on-demand deployment, in place of the model name above.
on_demand_model = "accounts/<your-account-id>/deployments/<your-deployment-id>"
A deployment that has scaled to zero returns a 503 while it scales up, so give the task retries.
Handle rate limits
A 429 means you’ve reached a rate limit on your Fireworks account. You can move to an on-demand deployment, but its requests still count toward account-wide limits, and a 429 there usually means the GPUs are saturated.
Route calls through the LLM Gateway
To add budgets, fallbacks, and guardrails, add Fireworks AI to the LLM Gateway using the same secret, then point your client at the gateway. See Choose direct calls or the gateway.