Renting GPUs vs buying tokens

What is actually worth renting, and from where

Re-verified on 2026-08-19 against live pricing pages and public catalogue endpoints. Rates on a marketplace move hourly, so medians are quoted rather than the single cheapest listing, which is usually one host in one region and gone by the time you read this.

Verified rates worth knowing

Two fees that do not appear in any hourly rate. OpenRouter adds 5.5 percent on card credit purchases with a $0.80 minimum, and its advertised cheapest endpoint is frequently deranked, meaning it will not actually serve your traffic. Moonshot quotes K3 excluding tax, which is added at checkout by jurisdiction.

The formula that decides it

A rented GPU bills 730 hours a month whether you send it traffic or not. So break-even does not depend on how fast the hardware is. It depends only on rent and the price of the tokens you would otherwise buy:

An 8x H200 node at $24.00/hr, the cheapest verifiable rate found, is $17,520 a month. Against a $4.00/M output API that is 4.4 billion output tokens before the node is worth owning. Against a $0.66/M API it is 26.5 billion.

For scale, a developer whose agent generates continuously at 50 tokens per second, eight hours a day, twenty-two days a month, produces 31.7 million output tokens. That is roughly 140x short of the friendlier figure. Even running flat out 24/7 with no pauses, one stream at 50 tok/s yields 131 million a month, still 33x short.

The entry ticket is a node, not a GPU

Every cheap-H100 headline is misleading for this workload, because a frontier open-weight model is a ~1T-parameter mixture of experts. At FP8 that is roughly 1TB of weights before any KV cache.

vLLM's own recipe for Kimi K2 at FP8 states the minimum deployment unit is 16 GPUs, not eight. And Kimi K3, at 2.78T parameters and 1,561 GB even in its native MXFP4 form, needs more than 1.5TB before KV cache. It does not fit an 8x B200 node.

The memory-density argument favours AMD hard. Two AWS p5.48xlarge nodes (16x H100) run about $80,358 a month. One 8x MI300X node with 2.4x the HBM runs about $17,462. That is a 4.6x spread before InfiniBand and egress.

Which open-weight models clear a frontier bar

Scores are the Artificial Analysis Intelligence Index, v4.1.1. Where a figure is an estimate rather than a measurement, it is marked, because the difference changes conclusions.

A caution on a comparison currently circulating: Qwen3.8-27B is genuinely remarkable, ranked first of 135 open-weight models in the 4B–40B class against a class median of 9. But claims that it beats Opus 4.6 rest on an Opus figure that Artificial Analysis flags as an estimate, measured in non-reasoning mode. Comparing measured against measured, it is 52 against Opus 5 at 63.

What a real 24/7 workload looks like

Most cost comparisons use a hypothetical user. This one uses a measured month: 1,039,011 assistant messages parsed from the session transcripts of a heavy agentic coding setup running fleets of parallel agents. The shape is the surprise.

Ninety-four percent of the volume is cache-read. That single fact reorders every provider comparison: rank by cache-read price, not by the headline input and output rates. Providers differ by more than 20x on that line while looking near-identical on their pricing pages. Anyone quoting dollars per million output tokens is pricing 0.6% of the bill.

Aggregate throughput is not what you feel

Every impressive tokens-per-second figure in the literature is measured at low per-user interactivity. The trade is steep, and it is measured on identical hardware, DeepSeek R1 at FP8 on H200:

Making the model 2.05x faster for the person using it costs 3.92x in aggregate throughput, and therefore 3.92x per token. A widely-quoted 13,860 tok/s peak on an 8x H200 corresponds to a mean time-per-output-token of 203 ms, which is 4.9 tok/s per user and unusable for interactive work.

Two published cost figures do not survive audit. The often-cited $0.21 per million for Kimi K2 counts only the decode GPUs and excludes the four prefill nodes; counting all 128 GPUs gives $0.284, and using today's cheapest on-demand H200 rate gives $0.435, about 2.07x the published number. A $0.33 per million H200 figure implies a GPU cost of $1.23/hr, which is an amortized ownership cost rather than a rental price.

The one figure that does survive is DeepSeek's own published production accounting: 226.75 nodes, 168 billion output tokens in 24 hours, $87,072 per day at a $2.00/GPU-hr internal cost, giving $0.518 per million output tokens at 20 to 22 tok/s per user. That is the floor a self-hoster is chasing, reached with 226 nodes and custom kernels.

Rent or buy

The 2026 GPU market is inflated. The RTX 5090 launched at $1,999 and retails around $4,530, roughly 2.3x list. That inverts the usual advice: buying is weaker than it was, renting stronger.

Two findings worth more than the prices. Enterprise clouds are the wrong channel for this workload: one major provider lists no consumer cards at all, and its cheapest part that fits a 27B model costs about 5x the marketplace rate for identical silicon. And a 3090 bought to run 24/7 has a 117-month breakeven against marketplace rates, where European electricity alone nearly matches the rent.

For the buy-it-outright case, a pair of 128GB unified-memory desktop AI boxes lands near $8,000 and runs a 167 GB mixture-of-experts model at a measured 41 tok/s single-stream, 62 to 75 with speculative decoding, and about 350 tok/s aggregate at 32 concurrent streams. Against equivalent-capacity rental that pays back in 3.3 months, and break-even utilization is only about 3.3 hours a day.

Quantization, and where the damage actually lands

Four-bit is not automatically degradation. It depends entirely on the reference precision the vendor shipped.

The trap is that post-training 4-bit is the only path that fits some models onto an 8x H100 node, which is exactly the configuration people price when they imagine self-hosting. Across 133 provider endpoints for twelve frontier models, 59 serve FP8, 23 serve FP4, 7 INT4, 3 MXFP4 and 38 do not declare. Roughly a quarter of frontier endpoints are 4-bit, and on aggregator routing, price sorting will silently send you to them. Pin providers explicitly.

What the adversarial audit removed

This study was produced by parallel research agents and then handed to a separate reviewer whose only instruction was to refute them, defaulting to failure on anything it could not verify against a primary source. It killed five claims. They are listed because a cost study that hides its corrections should not be trusted with your money.

One figure survives with a caveat attached rather than a correction. The claim that self-hosting Kimi K3 costs more than its own API rests on a published throughput number the auditor could not source. Both prices in that calculation are confirmed; the throughput is not. The direction holds regardless, because K3 needs ten to fourteen GPUs, but the exact figure should not be quoted as hard.

How to read this before spending

Method

Workload figures were produced by parsing the usage field of 1,039,011 assistant messages across 13,587 local session transcript files, covering 2026-07-18 to 2026-08-19. No personal or account-identifying data appears on this page.

Model sizes were computed by summing .safetensors byte counts from the Hugging Face model API, and architecture figures were read from each model's own config.json rather than from a summary. Prices were read from provider pricing pages and public unauthenticated catalogue endpoints on 2026-08-19. All cost arithmetic is shown inline so it can be checked.

Figures described as measured were produced by running code or reading a primary source. Figures marked as estimates were not. Five claims were corrected or removed by adversarial audit after first drafting, and all are disclosed above. GPU and token pricing in this market moves weekly; verify before committing money.

Related