AI and GPUs

AI and GPUs runs models and GPU workloads in your project: model endpoints that serve open models over an OpenAI-compatible API, serverless endpoints for your own GPU containers, GPU pods and notebooks, fine-tuning, training and batch jobs. It runs on your own GPU servers, on GPU servers you rent from Frazer, or on the Frazer fleet by the second, and one placement field picks which. Every resource is an API object, so the console, the frazer CLI, MCP and the Frazer app drive the same thing, and every change is a plan you can read before it runs.

Product overview: AI and GPUs.

How it works

AI and GPUs is one console area, AI and GPUs, with one noun per page. A model is weights; a model endpoint serves a model; a serverless endpoint runs your container; a GPU pod is an interactive container; a job runs to completion. Pick the page for the job you have:

You want to	Use	Console page
See what GPUs you have and what they cost	Capacity, quotas and spend	Overview
Serve an open model behind an OpenAI-compatible API	A model endpoint	Model catalog, Model endpoints
Run your own GPU code as an API or a job queue	A serverless endpoint	Serverless endpoints
Compare models on the same prompt	The playground	Playground
Use a raw GPU, or open JupyterLab	A GPU pod or notebook	GPU pods
Adapt a model to your data	A fine-tuning job	Fine-tuning
Train across several servers	A GPU cluster and a training job	GPU clusters, Training jobs
Process a file of requests overnight	A batch job	Batch jobs

A devbox with a GPU is where you write code; a GPU pod is where you run a container on a GPU.

Key concepts

Model: weights pinned to an immutable revision (a commit SHA, version ID or digest), from Hugging Face, an upload, a bucket or the output of one of your jobs.
Model endpoint: Frazer runs its inference engine for the model kind (chat and completion, embeddings and reranking, or GGUF files), tuned per model and GPU class, and serves an OpenAI-compatible API that scales to zero.
Serverless endpoint: your container on GPUs, as a job queue (queue) or an HTTP server (http), scaling to zero.
GPU pod: a container with whole GPUs, SSH and attached volumes. A notebook is a GPU pod with the JupyterLab preset.
GPU template: image, start command, ports, environment, secret references, volume mount and allowed GPU classes, for pods and serverless endpoints.
GPU class: one spelling everywhere, such as h100-80gb, h200-141gb, b200-180gb, l40s-48gb or rtx6000ada-48gb, with one price per class per region.
Placement: where a workload runs, set by one field on every compute resource (below).
Volume: network storage for pods, serverless endpoints and jobs, with one writer and many readers.

Where workloads run

Placement	What it is	Isolation	Billed
Your own servers (`placement: { pool: … }`)	GPU servers you joined to Frazer	Your organization only; MIG and time-slicing allowed inside it	A flat fee per GPU per month; everything on them is included
Rented from Frazer (`placement: { pool: … }`)	Whole GPU servers dedicated to your organization for a term, wiped with a certificate afterwards	As your own servers	The rental and the GPU fee; no per-second charge
Frazer fleet (`placement: { location: … }`)	Shared Frazer GPUs	A VM per organization with whole GPUs passed through	Per GPU-second by class

The deploy and create flows list every placement that fits, cheapest first, each with its price and the live count of free GPUs. Between organizations, Frazer only ever hands out whole GPUs.

How weights reach a GPU

An import resolves the model to an immutable revision, reads its config, license and gating, and downloads only the files the chosen format needs, with your own Hugging Face credential. Every file is stored with its SHA-256 in your organization’s model store. Safetensors only by default: pickle files are refused unless you opt in for that import. When a worker is scheduled, the model is prefetched to the server’s local NVMe, and the scheduler prefers servers that already hold it. Gated and private weights stay in your organization, and access is checked again on every pull.

Deploy a model

You need a project, GPU capacity in one of the three placements, and the frazer CLI signed in to your organization (CLI reference).

Check capacity. See GPUs per class on your own servers, rented servers and the fleet, with the project’s quota and queued work:
Terminal window
```
frazer ai overview
```
The console shows the same table on AI and GPUs › Overview › Capacity.
Connect Hugging Face if the model is gated. Open AI and GPUs › Integrations, choose Hugging Face, and sign in with OAuth or paste a fine-grained token. The connection belongs to your organization and is used only by the import step.

Plan the deploy, then deploy.

frazer ai endpoints deploy hf://openai/gpt-oss-120b --project acme-production \
  --gpu h100-80gb --min 0 --max 3 --dry-run

Plan: deploy hf://openai/gpt-oss-120b as endpoint gpt-oss-120b in acme-production
  + Model import   pinned to the current revision · safetensors only
  + Endpoint       gpt-oss-120b · kind llm · Frazer inference · OpenAI-compatible
  + Workers        1 × h100-80gb each · min 0 · max 3 · idle 10 min
  ✓ Where          fleet us-virginia1 · your pool sf-office-default has no free GPU
  ✓ Quota          gpu-h100-80gb within the project quota
    Cost           per GPU-second while a worker is ready · nothing at zero

Run the same command without --dry-run to deploy.

apiVersion: ai.frazer.ai/v1
kind: Endpoint
name: projects/acme-production/endpoints/gpt-oss-120b
spec:
  kind: llm
  model: hf://openai/gpt-oss-120b
  gpu: { class: h100-80gb, count: 1 }
  placement: { location: us-virginia1 } # or { pool: sf-office-default }
  scaling: { min: 0, max: 3 }

frazer apply -f gpt-oss-120b.yaml --dry-run

Run it without --dry-run to deploy. The endpoint is also a resource in the Terraform and OpenTofu provider, with the same schema.

PROJECT=projects/acme-production
curl -X POST "https://api.frazer.ai/ai/v1/$PROJECT/endpoints" \
  -H "Authorization: Bearer $FRAZER_TOKEN" \
  -d '{"model": "hf://openai/gpt-oss-120b", "validateOnly": true,
       "gpu": {"class": "h100-80gb", "count": 1},
       "scaling": {"min": 0, "max": 3}}'

validateOnly returns the plan. Send the same request without it to deploy.

Call the endpoint. The endpoint’s Overview has a Call it panel with its OpenAI-compatible URL and a ready-made request. Point any OpenAI SDK at that URL, with a token from the LLM gateway (frazer llm token prints a 1-hour token for you) or an endpoint key from frazer ai endpoints keys create:
```
import subprocess
from openai import OpenAI

token = subprocess.run(["frazer", "llm", "token"], capture_output=True, text=True, check=True).stdout.strip()
client = OpenAI(base_url=ENDPOINT_URL, api_key=token)  # ENDPOINT_URL: the URL from Call it
print([m.id for m in client.models.list()])
```
To call it by a route name, with budgets and request logs, go through the LLM gateway: every ready model endpoint appears there as a provider.
Compare it in the playground. Open the endpoint’s Playground tab and send a prompt, or open AI and GPUs › Inference › Playground to send one prompt to up to three targets side by side (model endpoints, temporary endpoints or LLM gateway models). Each answer shows its time to first token, tokens per second, tokens and cost, and View code shows the request it sent. Try on a catalog row creates a temporary endpoint that stops after one hour idle and opens it in the playground.

Each model endpoint runs the serving engine that measured fastest for its model, GPU class and speed tier. Frazer changes an endpoint’s engine only after conformance and accuracy checks and a 7-day soak, and one click returns an endpoint to the default engine. Every model endpoint shows and pins its engine, engine version, quantization, maximum context, prefix-cache hit rate and cold start p50 and p95. Every change creates a revision you can roll back in one click, and you can set a daily spend cap per endpoint.

Run your container as a serverless endpoint

A serverless endpoint runs your own container on GPUs and scales to zero. Choose a mode:

Mode	Contract
`queue` (default)	`POST /run` returns a job ID; `POST /runsync` waits up to 60 seconds, then returns the job ID; `GET /status/{id}`, `GET /stream/{id}`, `POST /cancel/{id}`, `POST /retry/{id}` and `POST /purge-queue`; a webhook when a job completes. Payloads up to 20 MiB inline, larger ones through a presigned bucket URL
`http`	Any HTTP server on one port, WebSockets included; requests go to the worker with the fewest in flight

Deploy from a repository. Frazer builds the image through the Frazer Reader GitHub App, scans it for secrets before the push, and lists up to three GPU classes in order, so a busy class falls back to the next:

CLI
YAML

frazer ai serverless-endpoints deploy --repo acme/transcribe --gpu l40s-48gb,rtx6000ada-48gb --mode queue --dry-run

apiVersion: ai.frazer.ai/v1
kind: ServerlessEndpoint
name: projects/acme-production/serverlessEndpoints/transcribe
spec:
  mode: queue # queue | http
  source: { repo: acme/transcribe, ref: main, template: gpuTemplates/whisper-worker }
  gpu: { classes: [l40s-48gb, rtx6000ada-48gb, a100-80gb], count: 1 }
  placement: { location: us-virginia1 } # or { pool: sf-office-default }
  workers: { min: 0, max: 5, keepWarm: 60s, scaleOn: { queueDelay: 2s } }
  executionTimeout: 10m
  jobTtl: 24h # counted from the start of execution
  resultRetention: 24h
  volumes: [{ volume: volumes/asr-models, mount: /models, readOnly: true }]
  spendCap: { dailyUsd: 80 }

frazer apply -f transcribe.yaml --dry-run

Submit a job and wait for its result (--sync), or leave out --sync to queue it and check its status:

frazer ai serverless-endpoints run transcribe --input @job.json --sync

Each release is an image digest. A new release replaces workers one by one; a worker on an old release finishes its job, then stops.

Worker states, each billed as shown:

State	Meaning	Billed
`Initializing`	Pulling the image and loading weights	No
`Idle`	Warm and waiting, inside the keep-warm window	Yes, shown separately as idle-warm
`Running`	Working on a job or request	Yes
`Draining`	Finishing a job before a release or a scale-down	Yes
`Throttled`	No GPU of the listed classes is free; counted as queue delay	No
`Unhealthy`	Crash loop; after 3 failures the release is halted and the previous one kept	No
`Stopped`	Scaled down	No

Jobs are Queued, Running, Completed, Failed, Canceled or TimedOut. The endpoint page shows queue depth, delay and execution time (p50, p95), cold starts, worker states, jobs per minute and cost per 1,000 jobs.

Batch jobs take an OpenAI batch JSONL file and run it against a model endpoint or a serverless endpoint within a 24-hour window, on spare capacity, with the output written to a bucket. Items that are not processed, and canceled items, are free.

Open a GPU pod or notebook

A GPU pod is a container with whole GPUs. Start one from a template and give it a maximum runtime:

frazer ai gpu-pods create --template pytorch --gpu h100-80gb --max-runtime 2h

Then frazer ai gpu-pods ssh opens a shell on it, frazer ai gpu-pods cp copies files to or from it, and frazer ai gpu-pods stop stops it.

SSH uses certificates from your organization’s SSH certificate authority, the same path as devboxes.
Ports are published on the apps domain behind Frazer sign-in.
Storage: attached volumes are kept; the 400 GiB scratch disk is erased when the pod stops. Stopping keeps storage only, and starting again places the pod on any server of the class in the region.
Notebook is the JupyterLab preset, with idle stop after 15 minutes. Any pod can set its own idle stop.
States: Creating, Starting, Running, Stopping, Stopped, Failed, Deleting.

GPU templates come in three scopes. Official templates (PyTorch, vLLM, SGLang, ComfyUI, JupyterLab, Axolotl) are rebuilt and tested every month on every GPU class. Organization and Personal templates are versioned, and you make them from a running pod (frazer ai gpu-templates create --from-pod vllm-eval), from a repository with a .frazer/template.yaml that is built and tested on every push, or from a devbox template image. A template can be shared with other projects of your organization.

Fine-tune and evaluate a model

A fine-tuning job adapts a base model to a dataset and produces a new model: an adapter, or full weights. Methods are SFT, DPO and GRPO; adapters are LoRA, QLoRA or DoRA, or you train the full model. Presets cover the common cases, and an Axolotl YAML file is accepted as is.

frazer ai fine-tuning-jobs create --base gpt-oss-120b --data ds/support@v3 --method sft --lora \
  --deploy-to gpt-oss-120b --if-beats support-v3

What happens:

The dataset is validated: schema, token count, a personal-data scan and a cost estimate.
The job is admitted to the project’s queue, and the base model is staged on local NVMe.
Training writes a checkpoint every N steps.
An evaluation compares the result with the base model on a held-out split, the benchmarks you chose, and an LLM judge. Results go to the project’s managed MLflow and show on the job’s Evaluation tab.
The output is a model in your catalog. With --deploy-to, the adapter is loaded on the running endpoint without a restart, but only if the evaluation passes the rule you set (here: it beats support-v3).

A failure caused by the platform is not billed and shows as a credit on the job.

Datasets are named, versioned pointers to bucket objects or to a Hugging Face or Kaggle dataset revision, with a schema kind (chat, preference, completion, images), rows, tokens and splits. Checkpoints of every job are listed with their step, metrics, size and state (staged, uploading, durable); you can resume a job from one or promote it to a model.

Train on a GPU cluster

A GPU cluster is a lease of whole servers in one network fabric. Lease 2 to 8 servers yourself (more on request), on your own servers, rented servers or the fleet, for a term from hours to months, and extend it in place. It holds a Kubernetes namespace with a job queue by default, or a Slurm flavor. A cluster costs the sum of its servers.

frazer ai gpu-clusters lease --servers 4 --gpu h100-80gb --term 72h
frazer ai train-jobs run -f trainjob.yaml --cluster asr-lease

Job shapes: torchrun, DeepSpeed, Megatron-Core and torchtitan jobs; Ray and verl jobs; or any JobSet. All are admitted all-or-nothing, placed by the fabric’s topology.
Preflight: a 2-minute NCCL all-reduce checks every node against its server class; a failing node is swapped for a spare.
Checkpoints save asynchronously to NVMe, then to object storage with a manifest; the job page shows the last durable one.
Failures: when a GPU or link fails (Xid, ECC or an InfiniBand link flap), the node is cordoned, the job fails fast and is queued again from the last durable checkpoint on a spare. frazer ai train-jobs resume --from-checkpoint restarts it by hand.
Stragglers: frazer ai train-jobs stragglers reports slow GPUs and ranks from clock skew and NCCL timing.

Access and audit

Who may create GPU compute: creating endpoints, pods and jobs needs the ai.editor role. Viewers see capacity, workloads and cost.
Every write is a plan. The console, CLI, API and MCP all return a plan (changes, quota impact and cost) before anything runs; production changes follow your organization’s approval rules.
Agents: the AI tools are MCP tools generated from the same API, read-only by default. An agent’s writes become plans that a person applies, spending above an agent’s limit asks the person, and every resource an agent creates records which agent created it and for whom. In the Frazer app, “deploy this model”, “run a pod” and “fine-tune on this data” each arrive as a plan card.
Credentials: Hugging Face, GitHub, registries, buckets, Weights & Biases and other connections belong to your organization and are stored in Frazer’s secret store. Each is used only by the step that needs it, and never copied into a serving container unless you mount it as a secret.
Licenses: the model page shows the model’s license; you accept it at import.
Isolation: on your own servers, only your organization runs. On the fleet, each organization gets its own VM with whole GPUs.

Limits and quotas

GPU quotas are per project and per GPU class (quotas/gpu-{class}), with a daily spend cap, under your organization’s policy. Only workers above zero, running pods and running jobs count against them: an endpoint scaled to zero frees its quota. Nothing decays silently because it is idle, and nothing is deleted for being idle.

Setting	Default
Model endpoint workers	min 0, max 3 (raising max is a quota check)
Model endpoint idle before scale-down	10 minutes
Cold request held for a worker (`scaleUpTimeout`)	5 minutes, then 503 with `Retry-After`
Model endpoint request timeout	10 minutes, at most 1 hour
Model endpoint scale signal	Queue depth of 8 per worker, or KV-cache use
Serverless endpoint workers	min 0, max 3
Serverless keep-warm window	60 seconds
Serverless scale signal	Queue delay above 2 seconds
Serverless execution timeout	10 minutes (5 seconds to 24 hours)
Serverless job TTL	24 hours, counted from the start of execution; queue TTL 24 hours
Serverless result retention	24 hours (up to 7 days), or write results to a bucket
Serverless GPU class fallback list	Up to 3 classes
GPU pod scratch disk	400 GiB, erased on stop
Notebook idle stop	15 minutes
GPU cluster, self-serve	2 to 8 servers
Storage after a cap stops compute	Kept at least 30 days

Request a quota increase from AI and GPUs › Overview › Quotas with a reason. Requests up to your organization’s policy are approved automatically; above it they go to approvals. Every plan says when a cap will be reached at the current rate.

Pricing

On your own GPU servers you pay a flat fee per GPU per month, and the endpoints, pods, jobs and notebooks on them are included; the console still shows GPU-seconds so each team sees what it used. On the fleet you pay per GPU-second by class, from the moment a worker is ready: starting, throttled, unhealthy and stopped workers are not billed. The idle-warm window costs the same GPU-second price and is shown separately. Batch jobs cost half the GPU-second price. Storage is billed per GiB-month and is never dearer while stopped, and there are no ingress or egress fees within fair use.

AI and GPUs	On your servers	On the Frazer fleet
Price	$49 per data-center GPU / month · $19 workstation GPU	Per GPU-second by class, see below

Every line, with the estimator: pricing.

Troubleshooting

Symptom	Cause	Fix
A request waits, then gets 503 with `Retry-After`	The endpoint was at zero and no worker became ready within `scaleUpTimeout`	Retry after the given time. For large models, set `min` to 1 so a worker stays warm
Serverless workers show `Throttled` and jobs queue	No GPU of the listed classes is free	Add a class to the fallback list (up to 3), or place the endpoint on your own servers
A release stops rolling out; workers show `Unhealthy`	The new image crashes; after 3 failures the release is halted and the previous one keeps serving	Read the Logs tab, fix the image, and deploy a new release
A deploy fails with `QUOTA_EXCEEDED`	The project’s GPU quota for that class is used	Scale an unused endpoint to zero, or request an increase; the error links to a pre-filled request
The class you want is sold out in a region	Every GPU of the class is in use	Choose Deploy when available to queue with a position, Notify me, another region, or a rented server
An import is refused	The repository has pickle files, or the model is gated and your Hugging Face connection has no access	Opt in to pickle for that import, or request access on Hugging Face and reconnect
New fleet compute is paused	The project’s cap or the endpoint’s daily cap was reached	Raise the cap; storage is kept and nothing is deleted
Files are gone after a pod restarted	They were on the scratch disk, which is erased on stop	Keep data on an attached volume

Ask the Frazer app “why is transcribe slow?”: it reads the queue, the worker states and the capacity, explains the cause with links, and proposes a fix as a plan.

Reference

CLI

Command	What it does
`frazer ai overview`	Show GPU capacity per class (your servers, rented servers and the fleet), project quotas, queued work and spend.
`frazer ai models import`	Import a model from Hugging Face or another source, pinned to an immutable revision.
`frazer ai models list`	List the models in the project.
`frazer ai models get`	Show a model's revision, files, license and lineage.
`frazer ai models export --oci`	Export a model as an OCI ModelKit to a model repository in Frazer Registry.
`frazer ai endpoints deploy`	Deploy a model as a model endpoint with an OpenAI-compatible API.
`frazer ai endpoints logs`	Show the logs of a model endpoint.
`frazer ai endpoints metrics`	Show the metrics of a model endpoint.
`frazer ai endpoints keys create`	Create a key for a model endpoint.
`frazer ai endpoints adapters add`	Load a fine-tuned adapter on a running model endpoint.
`frazer ai serverless-endpoints deploy`	Deploy your container from a repository as a serverless endpoint on GPUs.
`frazer ai serverless-endpoints run`	Submit a job to a queue-mode serverless endpoint, or wait for its result with --sync.
`frazer ai serverless-endpoints jobs list`	List the jobs of a serverless endpoint.
`frazer ai serverless-endpoints workers list`	List the workers of a serverless endpoint and their states.
`frazer ai serverless-endpoints purge-queue`	Remove every queued job of a serverless endpoint.
`frazer ai gpu-pods create`	Create a GPU pod from a template, with an optional maximum runtime.
`frazer ai gpu-pods ssh`	Open a shell on a GPU pod.
`frazer ai gpu-pods cp`	Copy files to or from a GPU pod.
`frazer ai gpu-pods stop`	Stop a GPU pod and keep its volumes.
`frazer ai gpu-templates create --from-pod`	Save a running GPU pod as a GPU template.
`frazer ai gpu-templates list`	List GPU templates (official, organization and personal).
`frazer ai volumes create`	Create a volume for pods, endpoints and jobs.
`frazer ai volumes attach`	Attach a volume to a running pod or job.
`frazer ai volumes detach`	Detach a volume while the workload keeps running.
`frazer ai volumes copy`	Copy a volume to another location.
`frazer ai fine-tuning-jobs create`	Start a fine-tuning job, and optionally deploy the result when it beats a model you name.
`frazer ai gpu-clusters lease`	Lease whole GPU servers in one fabric as a GPU cluster for a term.
`frazer ai train-jobs run`	Start a training job from a YAML file on a GPU cluster.
`frazer ai train-jobs resume`	Resume a training job from a checkpoint.
`frazer ai train-jobs stragglers`	Show the GPUs and ranks that slow a training job down.

All of them: CLI reference.

Events

Event type	When it is sent
`frazer.endpoint.ready.v1`	A model endpoint finished deploying and is ready to serve.
`frazer.endpoint.failed.v1`	A model endpoint failed to deploy or to stay healthy.
`frazer.finetune.completed.v1`	A fine-tuning job finished.
`frazer.job.completed.v1`	An AI job ran to completion.

All of them: Events reference.

AI and GPUs product overview
Deploy an open model quickstart
LLM gateway: one door to your endpoints and outside providers
Knowledge: company RAG on your embedding and rerank endpoints
Servers and pools: bring GPU servers and see their burn-in
Frazer app: deploy models and run pods from a chat