One safe door to every model.
Budgets that stop.
One OpenAI- and Anthropic-compatible endpoint for every person, devbox, agent and service in your organization. Clients ask for a route like balanced; the gateway holds the provider keys, reserves the budget before the call and checks the prompt on the way in. The models on your own GPUs sit behind the same door.
$ curl -i https://llm.frazer.ai/v1/chat/completions \
-H "Authorization: Bearer $(frazer llm token)" \
-H "content-type: application/json" \
-d '{"model": "balanced", "messages": [ … ]}'
HTTP/2 200
x-frazer-request-id: req_01J9W3K7ZP
x-frazer-route: balanced
x-frazer-route-step: 2
x-frazer-provider: bedrock
x-frazer-served-model: claude-sonnet-5-5
x-frazer-credential-source: org-key
x-frazer-cost-usd: 0.061
# step 1 · anthropic, your key · 529 overloaded
✓ step 2 · same model, labeled in the reply and the logs{
"apiKeyHelper": "frazer llm token",
"env": {
"ANTHROPIC_BASE_URL": "https://llm.frazer.ai/anthropic"
}
}# Any OpenAI SDK. OPENAI_API_KEY holds a Frazer key (frz_llm_…)
from openai import OpenAI
client = OpenAI(base_url="https://llm.frazer.ai/v1")
reply = client.chat.completions.create(
model="balanced", # a route, not a provider's model name
messages=[{"role": "user", "content": "Summarize incident #142"}],
)
print(reply.choices[0].message.content)Checked before it leaves.
Labeled when it comes back.
Change the model, not the code.
Clients ask for a route by name.
A route is a stable name with ordered steps, each a provider and a model. Changing the model behind balanced is one plan with a revision, and rolling it back is one click. No client changes a line.
Same model first
When a step fails, the next step serves the same model from another provider, with the data policy checked for every step when you save.
Another model only if you say so
Set
otherModelFallbackand every such reply carriesx-frazer-served-modeland a degraded mark in the logs.Your key never spends ours
A failing key of yours fails its step. Frazer's accounts answer only when the route lists them as a step.
Clear triggers
A step moves on after a 429, a 5xx, an overload, a lost connection or no first token in time, with one retry per step.
# frazer llm routes apply -f balanced.yaml --dry-run
name: balanced
dataPolicy: zdr
guardrailPolicy: prod-default
steps:
- provider: anthropic # org key, ZDR agreement
model: claude-sonnet-5-5
firstTokenTimeout: 20s
- provider: bedrock # your cloud account, same model
model: claude-sonnet-5-5
otherModelFallback: false
perKeyLimits: {rpm: 60, tpm: 400000}
cacheMode: provider-only$ frazer llm providers connect anthropic --account frazer --zdr
$ frazer llm routes apply -f balanced.yaml --dry-run
$ frazer llm keys create acme-billing-worker --routes fast,balanced --budget 200/month --rpm 60 --tpm 200k
$ frazer llm token # 1-hour token for the caller
$ frazer llm logs --status '!200' --since 1h
$ frazer llm usage --by key --month 2026-10Secrets stop at the door.
Checked before any provider or cache.
A guardrail policy is a named, versioned set of checks on a route or a key, and a key may only pick a stricter one. Any check can run in monitor mode first, logging what it would have blocked. Your organization's floor is set in Security Center; projects can only tighten it.
Built for a whole team.
People, boxes, agents and services.
OpenAI and Anthropic APIs
Chat Completions, Responses and Embeddings, plus Anthropic Messages passed through natively, so prompt caching and thinking survive.
Routes with revisions
Stable names such as
balancedorcoder-small. Change the model in one plan; roll back in one click.A key per owner
Services, people, devboxes, agents and knowledge bases. Boxes and agents get 1-hour tokens, never a stored key.
Budgets that stop
Organization, project, key and session budgets, reserved before each call and settled after.
Your GPUs as a provider
Every ready Frazer endpoint shows up. Its routes never leave your organization, and boxes share it fairly.
Usage you can trace
Every reply names its route step, served model and cost; usage rolls up by key, person, box, agent and PR.
Guardrail policies
Secrets blocked, personal data masked and restored, injection labeled, data policy per route.
ZDR mode and a private endpoint
Zero data retention per project, and
llm.{project}.internalinside your project's network.
Every key has an owner.
And a budget it cannot pass.
Who may do what
- Services
- A key delivered from Frazer Secrets by the cluster’s secret sync. It expires in 90 days; rotation keeps the old secret for 24 hours.
- People
- A 1-hour token from
frazer llm token, or a personal key for scripts that expires within 90 days. - Devboxes and agents
- Never a stored key: a 1-hour token per box, per person on a shared box, and per agent session.
- Agents
- Propose key changes over MCP; people apply them. A key never approves anything.
- Project admins
- Read stored bodies when a project keeps them, and every read is audited.
What is logged
- A usage row per request: key, route, step, provider, model, credential source, tokens and cost.
- Metadata only by default, kept 30 days. No prompt or reply bodies.
- Masked bodies for 7 days, or full bodies for 7 to 30 days where an organization policy allows it.
- A secret found in a prompt is hashed, never stored.
- ZDR mode keeps metadata only and denies body capture, the response cache and stateful provider features.
At the cap
- Budgets nest: organization, project, key, session. Periods are a day, a week or a month.
- Each call reserves its worst-case cost first. At 100 % a key blocks, or only alerts if you choose.
- Alerts at 50, 80 and 100 % to the key owner and project admins.
- A stream that was admitted is never cut off halfway.
- A single request above $2.00 is blocked by default, with the estimate shown.
- The organization kill switch stops every key of a kind in one write.
No fee on your own keys.
Pay for tokens, not for the door.
With your own provider keys the gateway, guardrails, metadata logs and metering cost nothing. Tokens through Frazer's provider accounts appear on your Frazer invoice, inside your project budgets and caps.
See every priceLLM gateway, answered.
What happens when a provider fails?
The route moves to its next step after a 429, a 5xx, an overload, a lost connection or no first token in time, with one retry per step. By default the next step serves the same model from another provider. Another model answers only if the route allows it, and the reply says which model it was. A failing key of yours never falls back to Frazer's accounts.
Do you keep my prompts?
Not by default. The gateway keeps metadata (key, route, model, tokens, cost and guardrail verdicts) for 30 days. Bodies are stored only when a project turns it on: masked for 7 days, or full for 7 to 30 days where your organization allows it, read by project admins with every read audited. ZDR mode turns capture off and allows only zero-retention routes.
How do Claude Code and other tools connect?
On a devbox, managed settings point ANTHROPIC_BASE_URL at the gateway and use frazer llm token as the key helper, so the box never holds a stored key. Claude Code always runs on your organization's own Anthropic key, or your own Bedrock, Vertex or Foundry credential, connected to the gateway; Frazer never bills Claude tokens used in Claude Code. Codex, Hermes, the Frazer Harness and OpenCode get the same helper. On a shared box each person gets their own key, so spend and audit stay per person.
Can five devboxes share one model on our GPU?
Yes. Deploy the model as a Frazer endpoint and publish it to boxes as a shared model; the gateway serves it as a route with a share of tokens per box and per person. When the GPU is busy, interactive requests go first and background ones queue (20 seconds by default), then get a 429 with Retry-After.
Which providers can a route use?
Anthropic, OpenAI, OpenRouter with your own key, any OpenAI-compatible URL such as Mistral, DeepSeek or Groq, your own Frazer endpoints, and Google Vertex AI, Amazon Bedrock and Azure OpenAI, each on your own cloud account. Routes, keys, budgets, guardrails and logs work the same for every provider. For org-only and air-gapped needs the gateway also runs as a data plane on your own servers, and an exact response cache is opt-in per route.