gpt-oss-120b: OpenAI's Open-Weight Model on a Single 80GB GPU

For years the practical answer to "can I run an OpenAI model on my own hardware" was no. gpt-oss-120b changed that on 5 August 2025: a reasoning model with published weights, a permissive licence and a deployment target of one data-centre GPU rather than a rack.

It is no longer new, and the open-weight field has moved a long way since. But its combination of licence, size and tooling is still unusual, and plenty of teams adopted it precisely because it was boring to deploy. This guide covers what it is, how to serve it without the most common mistake, and where its weaknesses are documented rather than guessed at.

The architecture in numbers

OpenAI's model card on arXiv gives the details. gpt-oss-120b is a mixture-of-experts transformer with about 116.8B total parameters and about 5.1B active per token. It has 36 layers, 128 experts per MoE block, and routes each token to the top four.

Attention alternates between a banded window of 128 tokens and fully dense layers, with grouped-query attention using eight key-value heads. The dense layers are extended to a 131,072-token context with YaRN. The tokenizer is o200k_harmony, with 201,088 tokens.

The ratio is the headline. About 5.1B active out of 117B total is very sparse, which is why throughput is good for the model's size: compute per token looks like a small model, while memory looks like a large one. You pay for the active parameters in compute on every token, and for the total parameters in memory before the first token is generated.

Why it fits on one GPU

The MoE expert weights were post-trained in MXFP4, a 4-bit format stored as packed values plus block scales. Everything else stays in BF16. The result is a checkpoint of 60.8 GiB according to the model card, and OpenAI states the model fits on a single 80GB GPU such as an NVIDIA H100 or AMD MI300X.

That is not a community quantisation with unknown quality loss. The model was trained to be run this way, so the 4-bit weights are the reference weights. The trade is that you need an inference stack whose MoE kernels understand MXFP4. vLLM, Ollama, LM Studio and Hugging Face Transformers are all documented on the model page, and OpenAI's GitHub repository includes a Triton implementation that runs the 120b model on one 80GB card.

Be careful with the reference PyTorch implementation in that repository. It is explicitly educational, upcasts to BF16, and needs at least four H100s. It is there to read, not to deploy. If you are sizing hardware from scratch, which open model fits your GPU covers how KV cache and concurrency eat into the headroom a checkpoint size suggests.

The harmony format is not optional

This is the mistake that produces the most confused bug reports. Both gpt-oss models were trained on OpenAI's harmony response format, and the model page says plainly that they should only be used with it, as they will not work correctly otherwise.

Harmony separates reasoning from the final answer into distinct channels and defines how tool calls are framed. If you serve the raw weights behind a generic chat template, you get output that looks almost right: reasoning leaking into answers, tool calls that do not parse, or responses that stop early. The fix is to use a server that implements harmony, which the documented stacks do, rather than writing your own prompt template.

Two related details from the same documentation. Reasoning effort is configurable as low, medium or high. And the model exposes its full chain of thought, which OpenAI says is meant for developers and is not intended to be shown to end users. Log it for debugging; do not render it in your UI.

What the benchmarks claim, and the weakness they admit

OpenAI's model card positions gpt-oss-120b against its own closed models: it surpasses o3-mini and approaches o4-mini on the main reasoning benchmarks, and does the same on coding and tool use. On HealthBench it outperforms o4-mini by a significant margin, according to the card.

The same card is candid about factual recall. Without browsing, gpt-oss-120b has a hallucination rate of 0.782 on SimpleQA against 0.750 for o4-mini, and 0.491 on PersonQA against 0.361. OpenAI attributes this partly to smaller models holding less world knowledge. The knowledge cutoff is June 2024.

Read that as a design constraint. The model reasons well over material you give it and is weaker at recalling facts on its own. Give it retrieval or a search tool for anything factual, and treat bare-question answers about people, dates and niche topics with suspicion.

Hosted, batch or self-hosted

You do not have to run it yourself. It is listed on OpenRouter at $0.15 input and $0.60 output per million tokens at the time of writing, with cached input at $0.075; rates vary by host because many providers serve the same weights.

There is also a :batch listing. It is the same model on a discounted asynchronous endpoint: you submit work and collect results later, in exchange for a lower rate. For evaluation runs, bulk classification or overnight document processing it is usually the right default. Batch API savings math covers when the latency trade pays.

Self-hosting makes sense when data cannot leave your network, when you want to fine-tune (the weights support it), or when sustained volume makes a dedicated GPU cheaper than per-token billing. The Apache 2.0 licence means none of those paths needs a separate agreement.

120b or 20b

The smaller sibling, gpt-oss-20b, has about 3.6B active parameters and runs within 16GB of memory. It is the better pick for local machines, latency-sensitive paths and high-volume simple tasks. The gpt-oss-20b guide covers it in detail.

Pick the 120b model when the task needs multi-step reasoning, reliable tool use across several calls, or the best quality you can get from an Apache-licensed OpenAI model on one card. If the job is content moderation against your own written policy, look instead at the purpose-built gpt-oss-safeguard-20b.

The decision rule: start with the 20b model on your real tasks, and move up only for the tasks where you can measure it failing. The 120b model costs more memory and more money per token, and it earns that only where reasoning is the bottleneck.