Most teams' access to language models starts with one API key in a `.env` file. Then a second team wants access, then a third, and then somebody ships an agent that calls a model a few thousand times an hour from a long-running process. By the time a platform team is asked to "own LLM access", there's a provider key in every notebook and CI job, a monthly invoice that nobody can break down, and no way to cut one service off without rotating the key everyone else uses.
Union now ships with the Inference Gateway, which gives the platform team one place to hand out that access and put limits on it. It runs in your Union data plane, in front of the hosted providers you already pay for and the open-weight models you serve on your own GPUs.
What's the Inference Gateway?
It's an OpenAI-compatible proxy that an admin deploys from the Union console, and it runs as a Union app in your data plane. Three things make it up:
- Providers: where calls go. That's either a hosted API (20+ of them, including OpenAI, Anthropic, Bedrock, Vertex AI, Gemini and Mistral) whose key you store once as a Union secret, or a vLLM or SGLang app you serve on Union.
- Virtual keys: the credential callers use instead of a provider key (`sk-un-…`). Each one has its own list of providers and allowed models, a budget in dollars per day, week or month, and a rate limit in requests per minute.
- One endpoint: `/v1/chat/completions`, the Responses API and `/v1/models`, with streaming. Models are addressed as `provider/model`, so `anthropic/claude-haiku-4-5` and `qwen/qwen3-8b` (a model you serve yourself) go through the same client. A request can also name fallback models to try if its provider fails.
Anything that speaks the OpenAI API can call it: the OpenAI SDK, LangChain, LiteLLM, curl, or a coding agent that reads `OPENAI_BASE_URL`.
Why we built it
Agents make key sprawl and opaque spend worse in four ways. They spend tokens unpredictably, they run as long-lived processes that each need a credential, nobody can say afterwards which of them spent what, and when the one provider an agent is hard-coded to has a bad hour, so does the product built on it.
A proxy with virtual keys covers most of that list, and plenty of teams already run one. What we kept hearing went a step further. One of the first teams on the gateway gave people across its org their own key with a budget, because frontier models aren't cheap and they wanted to see who was using how much. The same team serves an open-weights model on Union for work it can't send to a model vendor, and wanted that model behind the same endpoint and the same keys.
That second ask is why the gateway is part of Union. Union already runs the data plane, secrets, RBAC and model serving in your own cloud, so we built the gateway on top of them, and a model you serve on Union sits next to the hosted providers with the same keys, budgets and usage view.
Every agent gets a key with a budget
You issue a key per team, app or agent in the console (Settings → Inference Gateway → Virtual keys), store it as a Union secret, and call the gateway from a task the same way you'd call OpenAI. This task is adapted from the docs example, with spend tagged by a custom dimension:
The budget is checked when each request arrives, and spend is counted from the tokens in each response. Once the key has spent its daily budget, its next call comes back as a `402` instead of showing up as a line on next month's invoice, and every other key on the gateway keeps working. Each gateway replica gets a share of the key's budget from the control plane, so adding replicas doesn't raise what a key can spend.
A call to a model that isn't on the key's allowlist gets a `403 model_blocked`, and a missing or revoked key gets a `401`, so the errors tell the caller which limit it hit. The gateway's Usage tab shows requests, errors, tokens, latency and estimated cost per key and per model, over up to 31 days.
Your models and theirs behind one endpoint
You serve an open-weight model on Union with a few lines of Python, on the same cluster as the gateway:
Then you add it to the gateway in the console as a Union LLM (vLLM / SGLang app) provider named `qwen`, and set what it costs you per million input and output tokens. From then on, moving the triage task off a hosted model is a change to one string:
As long as the key's provider list includes `qwen` (an empty list allows every provider), the key, its budget, its rate limit and the usage view all stay the same, and the By model table shows the traffic moving. Because the app runs on the gateway's cluster, prompts sent to that model don't leave your cloud at all.
The cost you set is what puts the open model under the same budgets. A self-hosted provider with no cost set shows as unmetered, and its calls never count against a budget. Set a number (your GPU-hour cost divided by the tokens you actually serve is a fine start) and the open model is budgeted like any other.
Revoke one key, not the one everyone shares
Provider keys live in one place, as Union secrets that the gateway mounts, and callers never see them. Everyone else gets a virtual key, including the coding agent on an engineer's laptop:
If that key ends up in a screenshot or a public repo, you revoke it in the console, and the gateway stops accepting it in about fifteen seconds. Nobody has to rotate the Anthropic key that every other service depends on.
A new gateway allows anonymous access by default, so the virtual key is the only credential: anyone holding it can call the gateway, up to that key's budget and allowed models, until it's revoked. If you want callers to sign in to Union as well, turn anonymous access off, and laptops reach the gateway through `flyte proxy app`.
Where it runs, and what Union sees
The gateway pulls its policy from Union's control plane and sends back one usage record per request: the key, provider, model, token counts, estimated cost and latency. The record never carries a prompt or an answer (it does carry the `x-un-dim-*` values you send, so keep those to labels), and the control plane never connects in to the gateway. Calls to hosted providers go from your cluster to that provider, the way they would without a gateway. Calls to models you serve stay in your cluster.
If you need to check what goes out, you can register a pre-call and a post-call guardrail, which are Union apps you write and which apply to every key on the gateway. A pre-call guardrail can let a request through, block it with a message (for example when it finds something that looks like a Social Security number), or answer it from a cache, and a post-call guardrail can pass or block the answer.
Scope and what's next
These are the details to plan around:
- Fallbacks are an ordered `provider/model` list that the caller sends with each request (`extra_body={"fallbacks": [...]}`). The gateway walks it when a provider fails, but it doesn't add a fallback list on its own.
- Spend is attributed to the virtual key and to the `x-un-dim-*` dimensions you send. It doesn't trace to a project or run yet, and model spend isn't on the Cost page.
- Keys can't be edited. To change one, issue a new key and revoke the old one.
- Adding a hosted provider redeploys the gateway to mount its secret. A self-hosted provider's cost can't be edited; remove the provider and add it again.
- After a gateway replica restarts, the budget share it held stays counted as spent, so a key can get a `402` slightly before its actual spend reaches the budget.
- Guardrails can allow or block, and pre-call can answer from a cache. They can't rewrite a prompt or an answer, and there's one of each per gateway.
- The console shows one gateway per organization, and self-hosted models and guardrail apps need to run on the gateway's cluster.
Recap
- Budgets per key: each team, app or agent gets its own key, with a dollar budget per day, week or month, and its calls stop once it's spent.
- Your models and theirs: hosted providers and the vLLM or SGLang apps you serve on Union sit behind the same `provider/model` endpoint, keys and budgets.
- One key to revoke: provider keys stay in Union secrets, and a leaked virtual key stops working about fifteen seconds after you revoke it.
- In your data plane: Union's control plane sees usage records, never prompts or answers.
Try it
The gateway is part of Union (BYOC and self-managed). An admin deploys it from Settings → Inference Gateway, adds a provider backed by a Union secret, and issues a key:
The Inference Gateway docs cover deployment, providers, virtual keys and guardrails, and the examples call it from tasks and apps. If you'd like help putting your own models behind it, talk to the Union team.







