Token Cost Accounting Across Inference Stack Layers

Per-token cost is a sum of charges from four separate layers of the inference stack: compute, KV cache memory, storage I/O, and interconnect, each with its own physics and its own failure modes. It is a sum of charges from four separate layers of the inference stack: compute, KV cache memory, storage I/O, and interconnect, each with its own physics and its own failure modes. The invoice a team sees at the end of the month collapses all of that into a single line, and that collapse is what makes cost hard to control. Context length, output verbosity, cache misses, retries, and the underlying infrastructure choices behind the API call routinely add meaningful cost on top of the sticker rate. A useful illustration sits right in most pricing pages already: output tokens run substantially higher than input tokens across the market, a gap wide enough that ignoring it alone can throw off a cost forecast by a wide margin. None of that can be fixed by watching the invoice. It requires taking the number apart, layer by layer, which is what the rest of this piece does. This is a piece about infrastructure-layer cost attribution, not prompt engineering and not which model to pick.

How compute cost is structured across prefill and decode

Every inference request runs through two phases that behave like different workloads wearing the same GPU. Prefill reads the entire prompt at once and pushes it through a single dense forward pass. It is compute-bound: the matrix multiplies dominate, and cost scales with how long the prompt is. Decode is the opposite. It generates one token at a time, and each of those steps has to read the KV cache tensors for every token that came before it. That makes decode memory-bound rather than compute-bound. The GPU can be pegged at high cost with its compute cores sitting mostly idle, waiting on memory reads instead.

That duality matters for cost attribution because the same chip can be the bottleneck for two completely different reasons depending on which phase is running. Blame the wrong phase, and the fix goes to the wrong layer of the stack.

Utilization is the multiplier that sits on top of all of it. A GPU running at 30% utilization costs several times more per inference than one running at full utilization, since the fixed cost of the card gets spread across far fewer completed tokens. Batch processing addresses this directly: running several requests through decode together instead of one at a time can cut cost per output token by up to 30%. Continuous batching is the main tool that keeps utilization high during decode by filling GPU slots as requests finish rather than waiting for a batch to fully complete, and it deserves a fuller look, which it gets in the KV cache section below.

Cost per request is a reasonable metric day to day, but it hides prompt-length creep: a request that quietly grows from a modest few thousand tokens to many times that much context looks the same on that dashboard until the bill arrives. Cost per valid result is the sharper number, because it captures the retries that quality problems generate, retries that inflate prefill volume without producing anything a user actually keeps.

KV cache memory as its own cost layer: why it is the central bottleneck

The KV cache exists to stop the model from recomputing the key and value tensors for every prior token on every single decode step. Without it, generating a token deep into the sequence would mean reprocessing nearly all of the tokens that came before it from scratch, over and over. Its job is compute savings, but the price it charges in return is persistent, growing pressure on GPU memory. That's the trade the whole industry is now managing at scale.

The footprint grows linearly with context length, and according to Xu et al. (arXiv:2603.20397, March 2026), that growth imposes serious limits on GPU memory capacity, bandwidth, and throughput as context windows move from thousands of tokens into the millions. The magnitudes involved are not abstract. Spheron reports that a single Llama 3 70B request running a 128K context window needs about 42 GB of GPU memory just for its KV cache, on a card that only has 80 GB total. That leaves barely anything for the model weights themselves, and nothing at all for a second concurrent user. Pushing the same model out to a 1M-token context causes the KV cache to balloon to roughly 135 GB at FP16, approaching the roughly 140 GB the model weights themselves take up. At that point the cache is not a side cost sitting next to the model. It is bigger than the model.

The growth comes from several dimensions multiplying together, including the number of transformer layers, the number of KV heads, the head dimension, sequence length, and how many requests are being served concurrently. Pushing any one of those up causes the cache to grow with it.

Inference serving stacks are rebuilding, piece by piece, the memory management ideas that operating systems solved decades ago: paging, slab allocation, address translation, working-set tracking. The KV cache has effectively become the working set of the inference server, in the same sense an OS textbook uses that term for pages actively touched by a running process. The response to this problem is broadly grouped into five families: cache eviction, cache compression, hybrid memory solutions, novel attention mechanisms, and combinations of the above.

The five KV cache optimization techniques and what each one saves

PagedAttention, introduced in 2023 and shipped by default in vLLM, TensorRT-LLM, and other major serving frameworks, treats the KV cache the way an OS treats virtual memory: fixed-size blocks, allocated on demand, instead of reserving one long contiguous slab per request up front. That reservation approach wastes enormous amounts of memory to fragmentation and over-provisioning. PagedAttention's compute overhead is small, somewhere around 2 to 5%, in exchange for pushing effective memory utilization above 95%. SGLang takes a related but distinct approach with RadixAttention, its own prefix-aware KV cache, shipped by default rather than as an add-on.

Prefix caching is the highest-leverage lever available at the application layer, and it's also the one most teams get wrong on the first try. It works by hashing prompt prefixes and reusing the KV state on a cache hit, a mechanism both vLLM's prefix cache and SGLang's RadixAttention implement. Hit rates of 85 to 95% translate directly into cost savings on those hits, but that hit rate is only achieved through deliberate tuning, not merely because the framework supports the feature.

ProjectDiscovery's experience from April 2026 makes the failure mode concrete. Their system started at a 7% cache hit rate because dynamic working memory was living inside the system prompt, which meant nearly every step invalidated the cacheable prefix before it could be reused. Moving that dynamic content out of the prefix pushed the hit rate to 74%. Adding explicit cache breakpoints and deliberate time-to-live settings on top of that pushed it further, to 84%, and cut total LLM spend by 59 to 70% across 9.8 billion cached tokens. The lesson generalizes past this one case: prefix caching requires deliberate prompt structure. Turning the feature on in the framework is necessary but nowhere near sufficient.

Attention-layer compression works one level down, at the architecture itself. Multi-query and grouped-query attention reduce the number of KV heads a model carries, which shrinks cache size directly. Llama 3's GQA setup, with 8 KV heads, cuts cache size well below what standard multi-head attention would require. DeepSeek's MLA goes further, using low-rank joint compression of keys and values so the cache stores compact latent vectors instead of full KV tensors.

KV cache quantization attacks the problem from the number-format side. INT8 and FP8 reduce the bytes needed per cache element, and FP8 and lower-precision formats reduce the bytes needed per cache element, with lower-bit formats pushing the reduction further. Agrawal and Mayer (arXiv:2607.05399) make an important qualification here: compression ratio by itself is a weak predictor of end-to-end serving performance. In their benchmarks, KIVI4 held quality more stable than the alternatives, while SnapKV, a pruning-based method, delivered the strongest throughput specifically on long-context workloads. Which technique wins depends on the workload, not on the compression ratio printed in a paper's abstract.

CPU offloading and hybrid memory setups round out the list, and they deserve their own treatment in the storage section further down, since that's really where the tradeoffs live. Combined, these five families can collapse long-context inference cost by a factor of 4 to 40, and that wide range is the honest answer: it depends entirely on which techniques compound well on a given workload, and which cancel each other's benefits out.

Agentic workloads break the assumptions that KV cache optimization was built on

Almost everything described above assumes a chatbot pattern in which a prompt arrives once, the cache grows by appending to the end, and prefix caching plus forward-only eviction are correct because the content never moves. That assumption holds for a conversational turn. It falls apart for an agent.

Agentic workloads involve tool calls, retries, and trajectory pivots, and all three break the append-only assumption directly. Identical content can shift to a new position in the context between turns, which invalidates an exact-prefix cache even though nothing about the underlying content actually changed. Failed tool outputs get dropped or swapped out mid-context rather than appended at the end. The practical consequence, as Leyline (Ma et al., arXiv:2606.01065, May 2026) documents, is that production agent harnesses currently fall back to full re-prefill on every edit, paying the prefix-recomputation cost that caching was supposed to remove.

Leyline's proposed fix is a serving-side primitive: a declarative (span, replacement) directive that lets a policy state what needs editing while the kernel applies that edit in place, using a closed-form rotation correction on the positional encoding (RoPE) rather than re-running the affected tokens from scratch. In the paper's benchmarks, the splice kernel raises replay cache-hit rate by 11.2 percentage points and cuts latency by as much as 241 milliseconds, by reusing prefix work that a naive edit would otherwise throw away. A ten-line truncation rule built on the same interface lifts agentic solve rate by 14.3 percentage points on the debug-gym benchmark.

The broader implication for cost accounting is that agentic systems need per-turn cache cost modeling, not just per-request modeling, because the request-level number hides exactly where the re-prefill penalty is being paid. Cost per valid accepted result, again, tells more of the truth here than cost per request does.

Prefill-decode disaggregation as an architectural response to compute and memory cost interference

Running prefill and decode on the same GPU pool creates a resource fight. A prefill burst arriving mid-decode interrupts the steady token-by-token rhythm decode depends on, and under load that shows up as spikes in inter-token latency, the kind of stutter users notice directly.

Prefill-decode disaggregation is the architectural fix: prefill and decode run on physically separate GPU pools. Prefill nodes process the incoming prompt and then transfer the resulting KV cache across the network to a decode node, which then generates tokens without prefill traffic ever interrupting it. Every major serving framework, vLLM, SGLang, TensorRT-LLM, LMDeploy, and NVIDIA Dynamo, supports this pattern now, and it's running in production at providers including DeepSeek and Gemini.

The measured gains are large. SGLang reports several times higher decoding throughput on NVIDIA GB200 NVL72 clusters, using Mooncake and NIXL as the transfer backends that move KV cache between nodes. AMD's MORI-IO connector, tested in March 2026 and reported on the vLLM blog that April, achieves several times higher goodput on a single 8-GPU MI300X node compared to standard collocated serving.

None of that comes free. Disaggregation only works because KV cache tensors get shipped across the network from prefill to decode, and that network transfer becomes a new, explicit, measurable line item in the cost model, one that didn't exist when everything sat on one GPU. Active development across serving frameworks is aimed squarely at that cost: layerwise KV cache transfer through backends like Mooncake, and pipelined transfer approaches designed to overlap data movement with ongoing GPU compute, so the network hop stops sitting on the critical path.

The memory hierarchy as a cost gradient: HBM, host DRAM, NVMe, and the data that moves between them

HBM is the fixed point around which this entire cost structure is built. Model weights move from host DRAM into GPU HBM once, at startup, and then sit there for the life of the deployment, functioning as a stable, read-only store the GPU pulls from constantly. The memory wall, the point at which the GPU can't move weights and KV cache fast enough to keep its compute cores fed, is a bandwidth problem sitting inside HBM. It is not a FLOPS shortage.

Bandwidth varies a lot across current hardware. AMD's MI300X offers 5.3 TB/s of HBM3 bandwidth with 192 GB of VRAM. NVIDIA's H200 is 4.8 TB/s of HBM3e with 141 GB. The B200 pushes bandwidth up to roughly 8 TB/s of HBM3e, also with 192 GB. Dropping off HBM entirely and onto host system RAM connected through PCIe causes bandwidth to fall off a cliff, down to somewhere around 32 to 64 GB/s. Offloading model layers to that tier introduces real latency penalties on token generation. It has a place for KV offload in specific, latency-tolerant scenarios, but it cannot stand in for HBM anywhere throughput matters.

Supply is tightening the screws further. Micron's HBM capacity is reportedly sold out through calendar year 2026, and gaming GPU production is facing cuts of roughly 40% as fab capacity gets redirected. That shortage turns efficient use of the HBM already deployed, not just buying more or faster cards, into the actual engineering lever teams have available to them.

An intermediate tier is emerging to relieve some of that pressure. CXL memory and disaggregated NVMe-backed storage sit between HBM and the network, and NVIDIA's Inference Context Memory Storage Platform, announced at CES 2026, extends GPU KV cache out into NVMe storage directly. The software stack behind it, NVIDIA Dynamo, NIXL, and DOCA Memos, with Grove layered on top for topology-aware orchestration, handles routing KV cache blocks, promoting and demoting them between HBM and flash, and reusing cache across nodes. The target case is multi-node inference where the aggregate KV cache for a workload simply doesn't fit inside the cluster's combined HBM, no matter how it's arranged.

BOOST (arXiv:2609.13592, September 2026) adds a wrinkle worth understanding. It proposes reading host memory and HBM concurrently instead of staging data through host memory first and only then into HBM. Prefetching lets larger workloads fit that otherwise couldn't, but every byte pulled from host memory still has to be written into HBM somewhere, which eats into HBM bandwidth that would otherwise be serving compute. It's a bandwidth tax, and it needs to be accounted for as one, not treated as free capacity.

The underlying principle holds across all of it: every hop between tiers (HBM to host DRAM, host DRAM to NVMe, NVMe to remote storage) carries a bandwidth penalty, and that penalty appears as latency first and as cost per token second.

Interconnect cost: what KV cache transfer costs at the network layer

Once prefill and decode sit on separate machines, the KV cache produced by prefill has to physically travel to wherever decode is running. That trip is not free, and its speed has nothing to do with how fast either GPU can compute. It's bounded by the interconnect between them.

High-bandwidth interconnects are the engineering answer to that bottleneck. SGLang's Mooncake and NIXL backends move KV cache between nodes for disaggregated serving, and layer-pipelined transfer overlaps that RDMA movement with ongoing GPU compute so the network hop doesn't just sit there blocking progress.

Speculative decoding attacks the same cost from a different angle. A small draft model proposes several tokens ahead, and the larger model verifies them in parallel rather than generating one at a time, cutting latency by a documented 2 to 3 times and supported in both vLLM and SGLang. Fewer decode steps means less sustained KV cache reading per output token overall, which lowers the bandwidth burden on both HBM and, in disaggregated setups, the interconnect too.

The accounting consequence is that interconnect bandwidth in a disaggregated cluster is a shared resource under contention from every request moving through the system at once, and cost attribution has to include the transfer cost per KV cache block moved, separate from whatever each node's GPU compute costs. NVIDIA's ICMS cross-node cache reuse adds one more wrinkle on top: KV blocks can be pulled from a shared NVMe pool instead of recomputed from scratch, but that only pays off if routing and transfer latency stay low enough to beat recomputation. Storage speed and interconnect speed jointly decide whether reuse is actually the cheaper path, and neither one alone answers the question.

Provider billing models surface the same layers in a different form

Provider pricing tables are, in effect, a public admission of all of the above, itemized in the separate rates that follow. Anthropic lists separate prices for five-minute and one-hour cache writes, alongside a cheaper rate for cache hits. Google prices cached content and storage as distinct line items. OpenAI's API pricing table separates cached input from cache writes on eligible models. Each of these providers made a deliberate choice to expose the asymmetry between writing to a cache and reading from one, rather than folding it into a single blended per-token number.

The economics behind that split trace directly back to everything covered above: a cache write costs prefill compute and consumes KV cache memory the moment it happens, while a cache hit mostly avoids both. Whether that trade pays off for a given team depends on how often the same prefix gets reused before it expires, which is the same hit-rate question that determined whether ProjectDiscovery's prefix caching actually saved money or just added complexity. The invoice, in the end, is just the compute layer, the KV cache layer, the storage layer, and the interconnect layer, added together and printed as one number. Taking that number apart is not optional if the goal is actually lowering it.