The Cheapest GPU Is the One That Already Knows You

Twenty agent runs, ten GPUs, four in the afternoon. Run 17 finishes a tool call and comes back for its next step with a 30,000-token prompt: the system prompt, forty tool definitions, and everything the run has said, seen and decided since it started twenty minutes ago.

Twenty-nine thousand of those tokens are byte-identical to the prompt it sent four seconds ago.

GPU 3 still has them. Not the text, the KV cache: the keys and values the model computed the last time it read those exact tokens, sitting in HBM, already paid for. GPU 3 also has three requests queued behind it.

GPU 7 is idle.

Your load balancer sends the request to GPU 7, because that is what load balancers do, and a step that could have taken 64 milliseconds takes 2.1 seconds instead. The dashboard is green. You have just paid a GPU to recompute 29,000 tokens that already existed, in memory, one machine over.

The instinct that’s wrong here

I spent most of a decade building routing for streaming systems, so least-loaded is baked into me at the level of reflex. Pick the node with the shortest queue. Bound the tail. It works.

Except the one time it really mattered. At Zeta we pushed about a billion rows an hour from Kafka into ClickHouse, and what made that survivable was partitioning consumers by key, so each held a warm local aggregation for the accounts it owned. Someone always proposes round-robin across partitions, because it balances perfectly. It also means every consumer needs every account’s state, so a warm working set becomes a cold lookup on every single row. Perfect balance, catastrophic throughput.

That’s the same trade as the one above, and streaming people learned it years ago: when a worker holds state, where you send work is a question about the state, not the queue. What’s new is the exchange rate. In that pipeline a miss cost a network round trip. On a GPU, a miss costs a full prefill, and prefill is the most expensive thing the machine does.

So the question isn’t whether cache affinity matters. It’s how much cache there is to fight over, and that depends entirely on what you’re serving.

An agent run is ninety-seven per cent repetition

Chat traffic has decent prefix sharing. Agent traffic has absurd prefix sharing, and the reason matters.

Every step of an agent run resends the whole conversation. The system prompt doesn’t change. The tool definitions don’t change, and on a real agent they’re enormous, because each tool carries a name, a description and a JSON schema. The history only grows at the end. So step 17 is step 16’s prompt plus a tool result plus a model turn, and the first 96 or 97 per cent of the tokens are bit-for-bit what they were four seconds ago.

Multiply that by twenty concurrent runs, all sharing the system prompt and the tool block, and the cache isn’t a nice-to-have. The cache is most of the workload.

You can see the shape in real traces. Moonshot published an hour of live Kimi traffic alongside Mooncake, the platform that serves it: 23,608 requests, averaging 7,590 input tokens against 182 output tokens. Forty-one tokens read for every one written. They also report a wildly lopsided popularity curve, where over half of all cached blocks are never reused at all while certain blocks get hit tens of thousands of times.¹¹That distribution is the engineering problem in one sentence. If reuse were uniform you could hash requests to machines and go home. It isn’t, so hot blocks need replicating and cold ones evicting, and the router has to know which is which. Their honest ceiling for that workload is about 50 per cent reuse, even with infinite cache and infinite patience. On their own chat-to-a-paper product, it’s 90.

SGLang’s production numbers say much the same. After a month serving models in Chatbot Arena they measured a 52.4 per cent hit rate for LLaVA-Next-34B and 74.1 per cent for Vicuna-33B, which cut first-token latency by an average of 1.7 times (Zheng et al.).

And here’s the detail that makes agents worse than those averages, which I learned the hard way building voice agents at Rovers. A supervisor fanning out ten parallel branches from one shared prefix looks like the ideal cache case: ten requests, one prefix, nine free reads. Anthropic’s docs say plainly that it isn’t, because a cache entry only becomes available once the first response has begun, so if you need hits on parallel requests you have to wait for the first response before sending the rest. Fire all ten at once and you get ten cache writes and zero reads. Fire one, wait, then fire nine, and you get one write and nine reads at a tenth of the price.

That’s a racing bug, not a config bug, and it only exists because something underneath is keeping a tree.

The tree that remembers

The mechanism underneath is a prefix tree over tokens, and SGLang’s version, RadixAttention, is the cleanest published description of it. The tree maps token sequences to the KV cache tensors they produced, you walk it matching an incoming request, and whatever matches is compute you don’t have to do. Two details are worth stealing even if you never run SGLang.

The first is the eviction policy: LRU on leaves, not on nodes. Evicting the least recently used leaf means a shared ancestor can only go once every one of its descendants has gone, so the fat trunk in the figure above outlives the twigs by construction. Each node also carries a reference count of running requests, and only a node at zero is evictable, which stops a batch evicting its own working set out from under itself.

The second is the scheduling order. Given a queue, they sort by matched prefix length and run the longest match first, and the paper proves that longest-shared-prefix-first is the same thing as a depth-first traversal of the request tree, which is the optimal cache hit order offline. Their scheduler lands within 96 per cent of that optimum in practice.

The bookkeeping is nearly free, and they measured it rather than asserting it: on a benchmark with no reuse at all, 100 requests took 74.3 seconds and the tree operations took 0.2 of them. Under 0.3 per cent, which is why it’s on by default.

vLLM does the same job with hashes. Each block’s key hashes the parent block’s hash, the block’s tokens, and a set of extras: LoRA IDs, image hashes, and an optional cache_salt so one tenant can’t probe another’s cache by timing. Two things there bite people. Hashing the parent makes the key positional, so a change anywhere earlier invalidates everything after it. And vLLM only caches full blocks, so your match length is quantised down to a block boundary.

All of which is excellent, and completely useless if the request lands on a machine that holds none of it.

Do the arithmetic before you route

So do the arithmetic. It’s the whole argument.

Recomputing 30,000 tokens on one H100 takes about 2.1 seconds, pricing a forward pass at two FLOPs per parameter per token for a 14-billion-parameter replica at 40 per cent utilisation. Recomputing only the 900 new ones takes 64 milliseconds. Thirty-three to one, on one request, from nothing but where you sent it.

Now add the queue. Three requests are waiting on GPU 3, each around 8,000 input tokens, roughly Mooncake’s trace average. That’s 1.7 seconds of waiting in front of your 64 milliseconds of work. Total: 1.76 seconds against the idle GPU’s 2.12. The warm machine still wins, while being busier, and while looking worse on every dashboard you own.

Push the queue slider to four and watch it flip. At four queued requests the warm GPU takes 2.33 seconds and the idle one takes 2.12, and now the textbook answer is correct. That crossover is the honest part of this, and I’d rather you found it than took my word for it.

And it moves faster than you’d think. At three queued requests the warm GPU only wins if more than 80 per cent of the prompt is already cached. Drop the sharing to 70 and the idle machine wins outright, which for ordinary chat traffic is exactly the regime you’re in. Agents are unusual precisely because they live at 97.

But notice what the two numbers actually are. One is prefill work. The other is queue time on a machine that is also decoding for everybody else. Not the same currency, and pretending they are is the second mistake in this post.

Prefill and decode wear the same uniform

An H100 does 989 TFLOPS of dense bf16 and moves 3.35 TB/s of HBM.²²NVIDIA publishes 1,979 TFLOPS of bf16 for the H100 SXM, with an asterisk under the table reading “With sparsity”. The dense figure is half that. Probably the most misquoted number in this field, and the correction is sitting in a footnote on NVIDIA’s own page. Divide one by the other and the card wants 295 FLOPs of arithmetic for every byte it reads, or it sits idle waiting on memory.

Weights are two bytes per parameter and a forward pass is about two FLOPs per parameter per token, so every token riding along with a weight read contributes one FLOP per byte. You need roughly 295 tokens in a forward pass before arithmetic is the constraint at all.

Now look at what the two phases bring. A 30,000-token prefill arrives with 30,000 tokens in one pass: hopelessly past the line, compute-bound, saturated. DistServe measured the threshold directly and found that for a 13-billion-parameter model, a single 512-token sequence fully engages an A100 (Zhong et al.). One request. Batching prefill buys nothing and delays everybody in the batch.

A decode step brings one token per sequence. Far left of the chart, 0.3 per cent of peak, reading every weight in the model to do almost no arithmetic with them. The only way to make decode efficient is to batch hundreds of sequences so the weight read is amortised.

So one workload wants a batch of one and the other wants a batch of three hundred, and continuous batching puts them on the same card and asks them to share. A long prefill lands in the batch, every decoding request in flight stalls behind it, and the user watches the answer freeze mid-sentence.

The cleanest measurement of that is in the Mooncake paper, and it’s the number I’d put on a slide. Replaying real traffic, 20 machines each way, a 30-second cap on time to first token and 100 milliseconds per token after that: both systems met the first-token target on essentially 100 per cent of requests. On time-between-tokens, the disaggregated setup met it on approximately 100 per cent and the co-located one on 57. Same hardware count, same traffic. Forty-three per cent of requests stuttered, and not one of them was slow to start.

Chunked prefill is the usual compromise, and it helps, but the reason it isn’t free is lovely. Split a prefill into N chunks and each chunk has to re-read the KV cache of every chunk before it, so you move N + (N-1) + ... + 1 chunks instead of N. Quadratic memory traffic, to fix a problem caused by memory traffic.

Which raises an awkward question. If the cache is already being shuttled between phases, how much further can you push it before it stops being cache and starts being storage?

The cache is a storage tier now

Quite a long way, and the numbers are the surprising part.

That 29,000-token prefix is 4.8 GB of KV cache, at 160 KiB a token for a model with 40 layers and 8 key-value heads.³³Two bytes each for K and V, times 8 heads, times 128 dimensions, times 40 layers. Grouped-query attention is doing enormous work there: at 64 key-value heads it would be 1.3 MB a token. Nobody keeps that in HBM for long. Mooncake’s answer is to stop trying, and instead pool the CPU DRAM, SSDs and RDMA cards across the cluster into a disaggregated KV cache, with GPU memory as the top tier of a hierarchy rather than the whole of it.

Price the tiers against recompute, which is the comparison that matters.

• Over an 800 Gbps RDMA fabric, the kind Mooncake’s testbed uses, 4.8 GB moves in about 48 milliseconds. Recompute is 2,123.
• Off an ordinary NVMe drive at 7 GB/s, about 680 milliseconds. Recompute is still 2,123.

Read the second one again. It is three times faster to pull nearly five gigabytes off a disk than to recompute it on an H100. Even at a physically impossible 100 per cent utilisation the recompute takes 849 milliseconds and the disk still wins. The most expensive silicon in the building loses to a consumer SSD, because it isn’t being asked to fetch. It’s being asked to think.

That changes the routing question. Without a cache tier it’s a binary: the machine has your prefix or it doesn’t. With one, every machine can have it, at a price, and routing becomes arithmetic rather than lookup. Mooncake’s scheduler does exactly that, per candidate: transfer time plus queue time plus prefill time, smallest sum wins. Set the figure above to “Fetch over RDMA” and you’re running their comparison.

There’s a second effect. A request landing on a machine that lacks the prefix makes that machine pull the cache and keep it, so hot prefixes replicate themselves as a side effect of load.

Which is a good place to admit that this whole argument has a failure mode, and it’s the obvious one.

Where the balance actually sits

A router that only chases cache hits sends everything to one machine. The prefix everybody shares lives somewhere, that somewhere gets all the traffic, and you have built a very expensive single-threaded server with nine idle friends.

Nobody serious pretends otherwise. SGLang’s appendix on distributed routing, where a router keeps a meta-tree of which worker holds what, ends by admitting a trade-off between data locality and parallel efficiency, and calling better policies future work. That is the state of the art being honest about an unsolved problem, which I trust more than a paper claiming to have nailed it.

Mooncake picks a specific answer, a threshold you could implement this afternoon. Their scheduler compares the best prefix match anywhere in the fleet against the match on the instance it’s considering. Below a tuned constant, the local cache is good enough and the request stays put. Above it, the request goes elsewhere and drags the cache with it. One knob, manually tuned, with a footnote saying it ought to be adaptive.

And in the same algorithm, eight lines apart, is the sentence that made me want to write this. Prefill instances are chosen by cache-aware scheduling. Decoding instances are chosen by plain load balancing. The same scheduler, in the same function, running two different policies, because one phase cares where the cache is and the other genuinely doesn’t.

That’s the rule, and it generalises past any one system: route prefill by cache, route decode by load. Once the prefix exists, decode is a memory-bandwidth problem that wants the emptiest machine it can find, and affinity buys it nothing.

You can watch the same trade-off in public docs. OpenAI says cached states live on individual machines, that routing depends on machine load and a hash of the opening tokens including tool definitions, and that traffic above roughly 15 requests per minute on one cache key can trigger overflow routing. For higher volumes, partition across keys.

That is consistent hashing, with a documented shard size, exposed as an API parameter. A provider saying in public that affinity and balance fight, and that the ceasefire line sits around 15 requests a minute per machine.

The money makes the same shape. Both major providers bill cache reads at a tenth of the input rate and writes at 1.25 times, so on Claude Sonnet 5 that’s $2 per million input tokens against $0.20 for a hit.⁴⁴Figures as of publication, and the point of this note is that they move. Both providers have changed cache pricing more than once. Check the docs, not your memory, and certainly not mine. Which makes the discount sound like the point, until you notice the clock. The default entry lives five minutes, refreshed free on every use, so an agent that pauses for a slow tool call comes back to a cold machine. Caching isn’t a discount you switch on, it’s a property of your call pattern.

That last part you can fix without owning a single GPU.

What I’d do on Monday

• Measure your prefix hit rate first. Both providers report cached tokens in the usage block, and both report a zero that quietly means “too short to cache”. Under 50 per cent, you have a prompt-ordering bug rather than a serving problem, and no router will save you.
• Put the volatile bits last. A timestamp or a request ID near the top invalidates every token after it, because all of these schemes hash the prefix cumulatively. The most common own goal here, and free to fix.
• Serialise the first call in a fan-out. One request, wait for the response to start, then the other nine. It feels wrong and it is ten times cheaper.
• If you run your own fleet, route prefill by cache and decode by load. Two policies, not one policy applied gently to both.
• Price the alternatives in milliseconds, not hit rates. Transfer plus queue plus prefill, per candidate, smallest wins. A hit rate can’t tell you whether to wait. A millisecond can.
• Put a threshold on affinity and let it break. There has to be a load at which the router gives up on the cache. Choose that number deliberately, or meet it in an incident.

None of this is exotic. It’s the same lesson as sharding a stream by key, in a currency where a miss costs two seconds instead of two milliseconds. The classical answer optimises the variable that’s easy to see, and the one that matters is sitting in HBM on a machine you decided was too busy.

If you want the tokens rather than the placement of them, I’ve written about where the money goes and about what actually changed inside the models. This one is about a prompt costing 64 milliseconds or 2.1 seconds depending on a routing decision nobody in your org has opened in a year.

Sources

Papers

• Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W. and Xu, X. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, arXiv:2407.00079, 2024.
• Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C. and Sheng, Y. SGLang: Efficient Execution of Structured Language Model Programs, arXiv:2312.07104, 2024.
• Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X. and Zhang, H. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, OSDI, 2024.
• Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H. and Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP, 2023.

Documentation

• Anthropic. Prompt caching, Claude API documentation.
• OpenAI. Prompt caching, platform documentation.
• vLLM. Automatic Prefix Caching, design documentation.
• NVIDIA. H100 Tensor Core GPU, product specifications.

@article{ghosh2026cheapest,
  title = {The Cheapest GPU Is the One That Already Knows You},
  author = {Ghosh, Krish},
  journal = {krishghosh.com},
  year = {2026},
  month = {April},
  url = "https://krishghosh.com/writing/cheapest-gpu-knows-you"
}