Deploy an open model

At the end of this guide, gpt-oss-120b runs as a model endpoint in the project acme-production. Frazer runs its inference engine behind an OpenAI-compatible API, adds workers as requests queue up and scales to zero when the endpoint is idle. You try it in the Playground and call it from code.

You need:

• A project. This guide uses acme-production.
• GPUs: GPU servers of yours in a pool, or the Frazer fleet. A GPU server joins like any other server; its GPUs are found during inspection.
• GPU quota for the class in the project. An endpoint scaled to zero holds none.
• For a gated model such as Llama: a Hugging Face account with access to it.

Deploy the model

1. Install the CLI and sign in.

   curl -fsSL https://get.frazer.ai | sh
   frazer login

2. For a gated model, connect Hugging Face. gpt-oss is not gated; skip this step for it. For Llama, open AI and GPUs › Integrations and connect Hugging Face with OAuth or a fine-grained token. Frazer downloads gated weights with your own credential, keeps them in your organization’s storage and checks your access again on every new pull.

3. See where GPUs are free. In the console, open AI and GPUs › Overview and select the Capacity tab. For each GPU class it shows the GPUs free and in use on your own servers, the fleet’s free GPUs in each region (live), your project’s quota and the jobs queued. The plan in the next step shows the same for the class you pick.

4. Plan the deploy.

   frazer ai endpoints deploy hf://openai/gpt-oss-120b --project acme-production \
     --gpu h100-80gb --min 0 --max 3 --dry-run

   Output

   Plan  projects/acme-production/endpoints/gpt-oss-120b
     + Import     hf://openai/gpt-oss-120b · MXFP4
     + Endpoint   gpt-oss-120b · Frazer inference · 1× H100 80GB · min 0, max 3
     Where        fleet us-virginia1 · your pool sf-office-default has no h100-80gb free
     Cost         per GPU-second while a worker is ready · nothing at zero

   The plan lists every place the endpoint fits, your own GPUs first, each with its price and live free count. The import pins the model to an immutable revision and downloads only the files its format needs. The model’s license is shown, and you accept it with the import.

   To choose the place yourself, or to keep the endpoint in Git, describe it in a file and plan it with frazer apply -f gpt-oss-120b.yaml --dry-run:

   gpt-oss-120b.yaml

   apiVersion: ai.frazer.ai/v1
   kind: Endpoint
   name: projects/acme-production/endpoints/gpt-oss-120b
   spec:
     kind: llm
     model: hf://openai/gpt-oss-120b
     gpu: { class: h100-80gb, count: 1 }
     placement: { location: us-virginia1 } # or { pool: sf-office-default }
     scaling: { min: 0, max: 3 }

5. Deploy. Run the same command without --dry-run:

   frazer ai endpoints deploy hf://openai/gpt-oss-120b --project acme-production \
     --gpu h100-80gb --min 0 --max 3

   Output

   ✓ Imported      hf://openai/gpt-oss-120b
   ✓ Weights cached
   ✓ Ready
   → https://gpt-oss-120b-4f7a2c.frazer.run/v1
     OpenAI-compatible · key in the LLM gateway

   The endpoint’s page, under AI and GPUs › Inference › Model endpoints, shows its state, workers, engine and version, quantization, maximum context, and cold-start times.

6. Try it in the Playground. On the endpoint’s page, select the Playground tab and send a prompt; each answer comes with its metrics. AI and GPUs › Inference › Playground compares up to three models side by side, and View code shows the request it sent.

7. Call it from code. Any OpenAI client works; this one is the Python SDK (pip install openai). Point it at the endpoint’s URL, which Call it on the endpoint’s page shows, and authenticate with a token from the LLM gateway; frazer llm token prints a 1-hour token for you:

   hello.py

   import subprocess
   from openai import OpenAI
   
   token = subprocess.run(["frazer", "llm", "token"], capture_output=True, text=True, check=True).stdout.strip()
   client = OpenAI(base_url="https://gpt-oss-120b-4f7a2c.frazer.run/v1", api_key=token)
   
   model = client.models.list().data[0].id
   reply = client.chat.completions.create(model=model, messages=[{"role": "user", "content": "Say hello."}])
   print(reply.choices[0].message.content)

   A service that calls the model gets its own key (frazer ai endpoints keys create), with a budget and rate limits.

Serve Llama instead

Connect Hugging Face (step 2), then deploy the model by its Hugging Face ID, for example hf://meta-llama/Llama-3.3-70B-Instruct, with a GPU class it fits on. The Model catalog under AI and GPUs › Inference shows which classes each model fits on, and accepts a pasted Hugging Face URL to start an import.

Scaling and cost

• Scale to zero. With --min 0, workers stop after 10 minutes without requests. The next request is held while a worker starts; after 5 minutes without one it gets a 503 with Retry-After. For large models, keep one worker warm with --min 1.
• On the Frazer fleet you pay per GPU-second by class, from the moment a worker is ready. Starting, throttled, unhealthy and stopped workers are not billed.
• On your own GPU servers a flat fee per GPU covers endpoints, pods and notebooks; usage shows as GPU-seconds so each team sees its share.

Prices for both are on the pricing page.

Note

Each organization on the fleet gets its own virtual machine with whole GPUs. On your own servers, only your organization’s work runs.

Next steps

AI and GPUsEndpoints, serverless endpoints, GPU pods, fine-tuning and quotas.

LLM gatewayKeys, routes, budgets and logs for every model you call.

Bring your first serverJoin a GPU server of your own.

AI and GPUs on frazer.aiThe product page, with GPU classes and fleet prices.