Infrastructure Investment Decisions in Pre-Revenue AI Companies

The infrastructure calculus for pre-revenue AI companies has inverted. The primary capital decision is no longer training but inference, because inference is the daily load of a shipping product. If you get the KV cache architecture wrong early, the product can be economically dead before any revenue arrives.

Why pre-revenue AI companies face an infrastructure decision they cannot defer

Training is episodic. Inference is continuous. The cost structure of each is fundamentally different, and the commercial evidence confirms the shift. Fireworks now makes over $1 billion in annualized revenue and serves over 40 trillion tokens per day, as a pure inference business. Inferact, founded by people behind vLLM, raised a $150 million seed round, which is an unusually large seed for infrastructure whose core job is serving models efficiently. These signals point in one direction: inference infrastructure is where value accretes, and pre-revenue companies must design for the infrastructure of a shipped product, not the infrastructure of research.

The consequence is unavoidable. Infrastructure choices made before revenue arrives determine whether unit economics are viable when revenue does arrive. This is not optional timing. It is the structural condition under which every other infrastructure decision is made.

What the KV cache actually is and why its memory behavior drives everything downstream

The KV cache stores intermediate key and value vectors from previous tokens to avoid redundant computation during autoregressive generation. It is an inherent optimization of the Transformer architecture, not an optional feature. Its memory footprint is 2 × L × H × D × T × P, and it scales linearly with sequence length T. As context windows move from thousands to millions of tokens, the cache becomes the dominant memory consumer in the serving stack.

The concrete numbers make the problem vivid. For a LLaMA-3-70B model at a 128K context window, KV cache memory for a single request is approximately 42 GB, and it can exceed 100 GB across multiple concurrent requests. That surpasses the capacity of many high-end GPUs. And 128K-token context windows are production workloads now, not research benchmarks. Because of that linear scaling, the memory wall limits inference server throughput, and long-context tasks become physically impossible on standard hardware without a deliberate management strategy.

One might object that this is a model architecture problem, not an infrastructure problem. The answer is that the model is given. Context windows are expanding industry-wide regardless of which model a company runs, and the infrastructure must absorb the load that the model architecture produces.

How the GPU memory hierarchy turns KV cache growth into a tiered storage problem

The memory hierarchy moves from fast and small to slow and large: SRAM to HBM to DDR/DRAM to NVMe. HBM is the critical layer for inference because it stores both model weights and KV cache, making HBM capacity the immediate ceiling on concurrent users. The fastest tier is roughly one million times faster than NVMe storage. Because KV cache growth routinely exceeds HBM capacity, every production inference system must make explicit decisions about which KV blocks live where, and those decisions directly determine throughput and cost.

A three-tier KV storage architecture is now a defined engineering pattern. Hot KV blocks sit on GPU HBM at approximately 3.35 TB/s bandwidth, warm blocks on CPU DRAM at roughly 63 GB/s, and cold blocks on NVMe SSD at about 7 GB/s. At CES 2026, NVIDIA standardized KV cache offload to NVMe SSDs through its Inference Context Memory Storage Platform (ICMSP), extending GPU KV cache into NVMe-based storage. The three-tier architecture is now vendor-supported, not experimental. NVIDIA's software stack, including the DOCA Memos framework, the Dynamo distributed inference framework with KV cache offload, and NIXL, provides accelerated sharing of KV cache across nodes, working across HBM through CPU DRAM to direct-attached NVMe and networked external storage.

The data-path details matter. GPUDirect Storage eliminates the CPU bottleneck by allowing NVMe to communicate directly with GPU memory via DMA, skipping the traditional two-copy path through CPU RAM. Without GDS, CPU preprocessing limits GPU utilization materially. RDMA and InfiniBand are required for multi-node coherence. In MLPerf Inference benchmarking, GPUDirect RDMA on NVIDIA ConnectX-6 InfiniBand achieved full single-node performance on ResNet-50 workloads, and BERT workloads reached 90 to 96 percent of closed division performance. At the extreme, four GB300 NVL72 systems interconnected with NVIDIA Quantum-X800 InfiniBand achieved the largest scale ever submitted to MLPerf Inference, delivering 2.49 million tokens per second on DeepSeek-R1 offline. If you are evaluating storage vendors before revenue, you must now treat ICMSP compatibility as a first-class requirement.

Why precision format selection is an infrastructure financing decision, not just a model quality decision

BF16 is the most commonly used floating-point format in LLM training, widely deployed in production-scale models including Llama 3, Qwen, and Mistral. It is the default, and defaults have infrastructure consequences. For a 70B model, FP8 requires 70 GB while BF16 requires 140 GB. For a 70B model with a 128K context window, the KV cache alone can consume over 40 GB at BF16 on a single H100 80 GB, leaving little headroom for model weights and other working memory.

The precision decision is therefore a financing decision in disguise. FP8 KV cache cuts per-token KV memory in half compared to BF16. A company defaulting to BF16 KV format is implicitly doubling its NVMe offload requirements and halving the number of concurrent sessions supportable per GPU. Selecting the wrong approach for the workload wastes either precision or memory.

The five KV cache management strategies and which workloads each one actually fits

No single KV cache optimization technique dominates across all workloads. The choice of strategy is determined by context length, hardware constraints, and task type, and the wrong match costs throughput or accuracy. The research literature organizes KV cache optimization into five principal directions, and each has distinct deployment trade-offs.

The first is cache eviction, which removes tokens unlikely to be attended to in future steps based on accumulated attention scores or prefill-stage observation. H2O, the Heavy-Hitter Oracle, keeps a fixed-size cache budget and evicts the token with the lowest cumulative attention score, so it preserves critical context while discarding redundant information. SnapKV identifies important tokens during prefill by attention weight clustering and concatenates them with a recent observation window. Eviction fits high-throughput datacenter serving with long contexts where some token loss is acceptable.

The second is cache compression through quantization, and it reduces memory per token without discarding tokens. KIVI is tuning-free quantization to as low as 2 bits, applying per-channel quantization to keys and per-token quantization to values to exploit the different statistical properties of each. KIVI4 gives the most stable quality across models among tested compression methods, so it is the lower-risk default for quality-sensitive workloads. CaM, by contrast, yields large gains on selected QA workloads but has substantial workload sensitivity in both quality and realized compression ratio. Compression fits accuracy-critical reasoning, legal, medical, and code generation workloads, because token dropping is unacceptable there.

The third is hybrid memory solutions, which offload KV blocks to CPU DRAM or NVMe rather than evicting them. This preserves the full cache while extending effective capacity beyond HBM. It corresponds directly to the three-tier HBM to DRAM to NVMe architecture and requires NVMe bandwidth and GDS to avoid CPU bottlenecks. It fits long-context single requests and multi-turn conversations, because full history must be preserved there.

The fourth is novel attention mechanisms, and they replace standard O(T²) attention with architectures that reduce asymptotic complexity. These reduce cache size by design rather than by compression or eviction. They are model-side rather than system-side, so they matter to companies that control the model architecture and less to those running third-party open-source models. The fifth is combination strategies, which integrate multiple techniques such as eviction plus quantization plus offload to maximize efficiency across hardware tiers. Adaptive, multi-stage optimization pipelines are the most promising direction for production systems serving heterogeneous workloads.

Agentic workloads require a different primitive entirely. Standard eviction and prefix caching assume prompts arrive once and the cache grows append-only. Agentic LLMs break this by editing conversation history between turns: failed tool calls get retried, stale outputs get dropped. Leyline introduces a serving-side primitive for policy-driven cache editing via a declarative span-replacement interface. Its splice kernel lifts replay cache-hit rate substantially and cuts latency by up to 241 milliseconds by reusing prefix work that a prompt edit would otherwise destroy. A ten-line truncation rule routed through the Leyline interface lifts agentic solve rate substantially on debug-gym, because shorter contexts let the model attend to what actually matters. Leyline-style primitives fit multi-agent systems, coding agents, and any workflow where the serving system must accept directives from a policy outside the inference kernel.

Why early architectural commitment to a KV cache strategy is harder to reverse than a GPU procurement decision

KV cache architecture choices are non-fungible in a way that compute provisioning is not. Eviction strategy, memory tier layout, and precision format each embed assumptions about workload that become load-bearing once a product ships. The right choice depends on whether the product's workload has high prefix reuse, as in a system-prompt-heavy API, or low, as in open-ended generation. A company that picks the wrong model here and then tries to switch faces a re-architecture of its serving layer, not a configuration change.

The network architecture perspective sharpens the point. A company can move from spot to reserved GPU instances, change instance type, or shift cloud providers, and never touch application logic. The KV cache strategy is threaded through the inference stack and cannot be swapped independently.

Etched's case is the extreme version of this logic. Specialized transformer silicon justified by over $1 billion of customer demand under contract, and a $700 million raise at a $21 billion valuation after Jane Street tested the hardware. The pre-revenue company facing a KV cache decision is making the same bet, at smaller scale, whether it recognizes it or not.

What the build-vs-managed-inference decision actually turns on for a pre-revenue company

Managed inference is not a permanent architecture. It is the correct default until workload patterns are stable enough to justify the irreversible commitments that owned infrastructure requires. The argument against premature lock-in is strongest when product-market fit is unproven: the workload characteristics that determine KV cache strategy, context length distribution, prefix reuse rate, request concurrency, eviction tolerance, are only observable under real production traffic.

But managed inference defers, not eliminates, the KV cache architecture decision. NVIDIA's ICMSP, Dynamo, and NIXL are vendor-supported primitives, meaning a pre-revenue company can adopt a well-supported three-tier architecture without building it from scratch. Open-source serving stacks such as vLLM and SGLang expose KV cache configuration directly. The right question is not build or buy. It is which KV cache assumptions are embedded in the infrastructure being used, and whether they are compatible with the workload the product will generate.

How storage infrastructure for KV cache — NVMe throughput, RDMA, and lossless compression — determines whether the economics close

The storage layer is where KV cache strategy meets unit economics. NVMe throughput, interconnect bandwidth, and compression ratios determine how many concurrent sessions a given hardware configuration can support, so they determine whether the cost per token is viable before revenue arrives. The bandwidth gap between HBM at 3.35 TB/s and NVMe at 7 GB/s is nearly 500x. KV block placement and eviction decisions must account for this gap, because a cache miss that hits NVMe instead of HBM is not a small penalty. It is a qualitative change in latency.

GDS eliminates the CPU bottleneck in the NVMe path by allowing direct DMA between NVMe and GPU memory. Without it, the CPU becomes the throughput ceiling for cold KV block retrieval. RDMA is required for multi-node KV sharing. For workloads where lossless reuse is required such as legal, medical, and code, this is not optional. It is the difference between NVMe offload being viable or not.

The emerging architecture treats KV cache storage and reuse as a content-distribution problem at internet scale. DeepSeek-V3.2 to DeepSeek-V4-Pro reduced KV cache for 100,000 tokens from 9 GB to 1 GB. Transferring a 1 GB KV cache over a 10-Gbps link can be faster and cheaper than recomputing it. A pre-revenue company that treats NVMe selection, RDMA connectivity, and KV compression as procurement line items rather than architectural decisions is likely to discover, under production load, that its storage layer is the bottleneck it did not plan for. Purpose-built storage infrastructure for LLM inference, sized for KV offload bandwidth, ICMSP-compatible, RDMA-capable, with lossless compression in the data path, is what converts a correct KV cache strategy into realized throughput. The strategy and the storage must be co-designed.