Running Frontier Models on Rented GPUs: What It Really Takes

Open weights plus a rented GPU looks like an obvious arbitrage. The weights are free, hourly compute is cheap by the hour, and API pricing per million tokens seems high by comparison. Then you try to fit the model.

The gap between that plan and a working deployment is mostly arithmetic, and it is worth doing before anyone provisions anything.

Total parameters, not active ones

The first and largest misunderstanding is what a mixture-of-experts model needs in memory. MoE architectures activate a fraction of their parameters per token, and that fraction determines compute cost, not memory footprint.

You must hold every parameter resident, because routing decides which experts are used per token and any of them may be needed at any moment. DeepSeek V4 Pro activates around 49B parameters from a 1.6T total; the memory requirement is set by the 1.6T. Kimi K3 activates 104B from 2.8T total, with weights around 1.6TB on disk.

That single fact eliminates single-node deployment for the frontier open-weight models. No rentable single accelerator holds 1.6TB of weights, so these are multi-node deployments with fast interconnect between nodes. Mixture of experts explained covers why the architecture trades memory for compute this way.

Dense models behave differently and more predictably. Qwen 3.6 27B is dense, which means memory and compute scale together and a single accelerator is genuinely sufficient. It reports 77.2 percent on SWE-bench Verified, which makes it the reference point for what fits on one card.

Weights are not the whole memory budget

Even with the weights resident you are not done. The KV cache grows with context length and concurrency, and at long contexts it becomes a substantial fraction of memory in its own right.

This is where 1M context windows get expensive in a way that price lists do not show. Serving a handful of concurrent requests at very long context can require more cache memory than a naive plan allocated for the entire deployment. KV cache explained covers how the footprint scales.

Plan the memory budget as weights plus cache plus activation overhead plus headroom, then size against your intended concurrency and context length rather than against a single-request demo. The demo always fits; production frequently does not.

Quantisation buys headroom, unevenly

Reducing weight precision is what makes most rented-GPU deployments viable at all. Eight-bit is close to lossless for practical purposes. Four-bit saves substantially more and costs noticeably more quality.

The important caveat is that the loss is not uniform. Quantisation tends to preserve conversational fluency while eroding precise multi-step reasoning, which is exactly the capability coding work depends on. A quantised build can pass casual testing and fail on real tasks. Quantised model quality loss covers evaluating the build you actually intend to run.

Even at four bits, a 1.6T-parameter model does not become a single-GPU proposition. Quantisation moves you between tiers of hardware; it does not collapse the tier structure.

Utilisation is the whole economic argument

A rented GPU costs the same at three in the morning as at three in the afternoon. Per-token pricing costs nothing when you send nothing. That difference, not the hourly rate, decides the comparison.

Typical developer-tool traffic is extremely bursty: concentrated in working hours, in one or two time zones, with long idle stretches. A deployment sized for peak concurrency sits mostly idle, and the effective cost per token is the hourly rate divided by the tokens you actually generated, which is a much worse number than the one in the plan.

Compute the crossover honestly. Take your real monthly token volume, price it at API rates, and compare against twenty-four hours a day of rental for however many nodes you need — not against the hours you expect to use. If you cannot keep the hardware busy, the arbitrage does not exist. Self-hosting versus managed API cost works through the arithmetic.

The costs that never make the spreadsheet

Engineering time is the largest omitted item, consistently. Someone provisions the nodes, configures the serving stack, tunes batching, handles out-of-memory failures, upgrades the framework, and is paged when it breaks at two in the morning. That is a recurring salaried cost, not a one-off.

Cold start is the second. Loading a terabyte of weights across nodes is measured in minutes, not seconds, which means you cannot scale to zero between bursts and you cannot recover from a node failure quickly. Any autoscaling plan has to account for it.

Model churn is the third. New models ship every few months, and each evaluation means a redeployment, a re-tune and a re-measurement on hardware you are paying for throughout. Teams underestimate how often they will want to do this.

Then the ordinary infrastructure tail: monitoring, log retention, network egress, and a security posture for a service that is now yours to defend. Security review for open-weight models covers that part.

When renting genuinely wins

Sustained high utilisation is the clearest case. If you can keep the hardware busy around the clock — a continuous batch pipeline, a large user base spread across time zones — the economics flip decisively and per-token pricing stops being competitive.

Data residency and air-gapped environments are the other. Where tokens legally cannot leave a jurisdiction or a network, the cost comparison is irrelevant and the question becomes which model fits on hardware you are permitted to use. Choosing a model for self-hosting covers that shortlist.

Predictable fixed cost is a weaker but real argument. Some organisations prefer a known monthly figure to a variable one even at a higher total, because it is easier to budget and impossible to blow up through a runaway agent loop.

A sensible sequence

Start on an API and measure a full month of real token volume, split by input and output. You cannot size anything without that number and most estimates are wrong by a large factor.

Then check what fits. If your quality floor is met by a dense model in the 27B range, single-GPU rental is a real option and worth pricing. If it requires a frontier MoE model, you are pricing a multi-node cluster and the bar is much higher.

Finally, compute the crossover with engineering time and full-time rental included, and require a comfortable margin rather than a marginal win. A deployment that is ten percent cheaper on paper and adds an on-call rotation is not cheaper.