Managed API Providers for Open-Weight 70B Models Compared

For open-source LLM inference stack benchmarking and optimization, the decisive comparison is how each system handles decode bandwidth, KV cache residency, and concurrency. Those properties explain the trade-offs in how systems are built, but on their own they do not show that open-weight inference is faster or cheaper than inference from proprietary-model providers.

Why 70B models create a distinct infrastructure problem that smaller models do not

Autoregressive inference puts two different demands on hardware. Prefill processes the prompt, is compute-bound, and parallelizes well across GPUs. Decode generates one token at a time and is memory-bandwidth-bound, spending much of its time waiting for weight reads. More compute alone does not remove that bottleneck. At 70B scale, the mismatch forces consequential choices about memory, hardware placement, and how requests share resources.

A model of this size in FP8 occupies tens of gigabytes of GPU memory on a multi-GPU server, leaving the remainder for KV cache. Long contexts make that remaining space scarce. The BF16 KV cache for a single in-flight request can take up most of a high-end GPU’s memory. Concurrent requests therefore compete for both bandwidth and residency, even when the server has substantial aggregate memory. These limits set how fast tokens arrive, how many users can be served at once, and how much it costs to keep that service running.

How KV cache management became the central design problem in 70B inference

The KV cache avoids recomputing past token representations at every decode step, but its usefulness depends on how efficiently the serving engine stores it. vLLM’s PagedAttention addresses the foundational problem of fragmentation through OS-inspired paging, achieving near-zero wasted space. Allocation is only the first layer, however. Radix tree-based prefix sharing lets concurrent requests reuse cached representations of common prompt prefixes. This reduces redundant storage and computation rather than merely packing separate copies more tightly.

Compression addresses the next constraint, which is the size of the useful data itself. FP8 KV quantization nearly doubles effective memory capacity per token relative to BF16, while halving block size and the bandwidth needed to move blocks between tiers. Smaller blocks also make offload more practical. On Blackwell hardware, NVIDIA’s NVFP4 format provides 4-bit KV storage that is 75% smaller than BF16 and 50% smaller than FP8. More hot blocks can therefore remain in GPU memory before eviction becomes necessary.

Paging cannot remove duplicated prefixes, and prefix sharing cannot shrink every unique request. Quantized storage reduces the footprint, while eviction policy determines which blocks retain scarce GPU residency. The notes have no separate lossless-compression measurements that would show a further gain. A provider’s cache format, reuse strategy, and eviction policy together set the practical limits on context capacity and concurrent service.

What actually separates provider architectures — three different answers to the memory-bandwidth problem

The mainstream approach maximizes HBM bandwidth on GPU clusters. Together AI, Fireworks AI, and Inference.net follow this path, with differentiation coming from cache management, batching, and attention kernels. Hardware provides substantial room for those optimizations. A B200 has 180 GB HBM3e at approximately 8 TB/s memory bandwidth, with native FP4 support enabling roughly double the FP8 throughput for compatible models. Nevertheless, concurrent long-context requests can quickly fill the KV working set left after model weights.

Specialty silicon instead puts decode data in on-chip SRAM. Groq’s LPU provides 500 MB per chip at 150 TB/s, while Cerebras’s CS-3, using WSE-3, provides 44 GB at 21 PB/s on a wafer-scale chip. The trade-off is capacity, because SRAM is fast but comparatively small. Such systems offer strong decode performance but face KV working-set constraints that HBM-based systems handle more gracefully.

The third approach separates hardware by workload phase and extends storage across memory tiers. Rather than asking one system to serve both phases equally well, disaggregated serving assigns prefill and decode to hardware suited to each. This changes both the architecture and the scheduler. Its success depends on moving KV state between those systems without surrendering the utilization gains to transfer delays. That boundary is the central issue in the production deployments that follow.

Disaggregated prefill and decode in production — what it achieves and who has deployed it

AWS and Cerebras announced their Amazon Bedrock architecture on March 13, 2026, with AWS Trainium handling prefill and CS-3 systems handling decode. Amazon Elastic Fabric Adapter connects the two phases. Cerebras’s vendor-reported result was “5x more high-speed token capacity in the same hardware footprint.

SambaNova demonstrated live disaggregated inference at COMPUTEX 2026 using NVIDIA B200 for prefill and SN40 RDU for decode. Artificial Analysis verified the configuration at 2× the speed of a B200-only setup. Together became the first commercial customer, and the architecture was live at Vector Core Compute’s data center. The next generation remains a projection. SambaNova’s SN50, targeting production in H2 2026, provides 5× more compute and 4× more network bandwidth than SN40, with a projected 10× throughput improvement when combined with B300 prefill.

Both configurations make KV transfer a first-class design problem. At long context, one request’s cache can span many gigabytes, requiring line-rate movement, predictable sub-millisecond tail latency, and avoidance of host CPU overhead. The sub-millisecond target does not apply to the time it takes to transfer the entire cache. NVIDIA’s Inference Context Memory Storage Platform, announced at CES 2026, extends this design through BlueField-4 DPUs that manage cache movement off the GPU compute path. NVMe supplies an overflow tier for evicted blocks while GPU SMs remain focused on matrix math.

How the KV cache hierarchy extends to NVMe and what that enables for long-context serving

Eviction from HBM does not have to mean discarding reusable state. An NVMe tier can keep KV blocks, but how much that helps depends on how fast they return when needed. Capacity alone is not sufficient. If retrieval stalls decode, inexpensive storage can become an expensive interruption to GPU work. Engineers must assess whether offload preserves useful state more economically than recomputing it without disrupting token production.

Compression changes that calculation across the hierarchy. Writing blocks in FP8 instead of BF16 lets NVMe hold twice as much KV storage and cuts the I/O bandwidth needed by half. The storage tier becomes more practical as a result. Its remaining constraint is predictable read latency, with the design goal of keeping retrieval below the time the GPU would spend on the next attention matmul.

What MLPerf benchmarks reveal about 70B inference throughput at the hardware level

MLPerf provides a controlled basis for examining hardware-level inference throughput. The supplied material has no 70B MLPerf inference results, so it gives no basis for comparing the hardware architectures above by the numbers. Neither the reported Bedrock gain nor the verified B200-and-RDU comparison should be relabeled as an MLPerf result. They answer different questions under their respective configurations.

That distinction limits what can be concluded about managed APIs. Hardware throughput establishes scale context, while a deployed service also reflects batching, cache behavior, concurrency, and available context capacity. Those are separate levels of evidence. The material also has no matched benchmark against proprietary-model providers that would show open-weight inference to be faster and cheaper. The evidence shows how different memory designs deal with different bottlenecks, which limits how broadly the findings apply.

How do managed 70B providers compare on latency, throughput, and cost?

Groq’s LPU architecture is a fit for latency-sensitive voice agents, real-time chat, and interactive interfaces. Its strength is bandwidth-bound decode. The supplied catalog includes GPT-OSS 20B, GPT-OSS 120B, and Qwen 3.8 27B, so architecture alone does not establish availability of a particular 70B endpoint. Very long contexts at high concurrency remain the structural trade-off.

Cerebras applies the SRAM approach at wafer scale, making it a fit for workloads requiring very high token throughput. The Bedrock deployment also gives AWS-based teams a disaggregated serving route. Catalog availability remains a separate consideration. Its free tier includes GPT-OSS-120B and Qwen 3.8 27B, while Llama 3.3 70B and Qwen3 235B were removed from the public catalog in early-to-mid 2026.

Together combines GPU software optimization with the commercially deployed disaggregated configuration described above. Its catalog spans 200+ models across text, image, video, audio, and embeddings, including Llama 4 Maverick. Together’s catalog covers more than the 70B category, and its fit is high-volume open-weight inference at low per-token cost, particularly where teams also need a broad catalog or very long contexts. Llama 4 Maverick has a very large context window in the supplied offering.

Dedicated rollout inference supports reinforcement-learning pipelines. Compliance is another selection criterion, with SOC 2 and HIPAA listed. Its prices are $0.10–$3.00 input / $0.20–$15.00 output per million tokens depending on the model, while size-tier models have one flat rate for both input and output. Those rates should be checked against the provider’s current pages before a team commits.

It fits cost-sensitive, high-volume work where latency is not critical. SambaNova’s direct API addresses a different reuse pattern through SN40’s SRAM, HBM, and DDR tiers, with DDR supporting lower-cost prompt caching across requests. That is why it is presented as a fit for multi-turn conversations and long sessions. The API has trial credits, works with another vendor’s API, and has a large context window, while the SN50 projections are still separate from how deployed systems perform.

Scaleway, OVHcloud AI Endpoints, and IONOS AI Model Hub form the European sovereign-provider group in this comparison. They are presented as a fit for EU-regulated workloads, GDPR-sensitive data, and work with hard residency requirements. Here, location can precede performance in the decision. IONOS specifies stateless inference with processing exclusively in Germany. That policy belongs to IONOS and should not be assumed to describe the other providers.

How to match your workload’s actual bottleneck to the right infrastructure approach

The decision begins with the constraint that binds first. For interactive workloads, the time to first token and the time it takes to deliver later tokens are separate concerns, because prefill and decode need different things from the hardware. SRAM architectures directly address the latter’s bandwidth bottleneck. Groq and Cerebras are therefore relevant candidates for latency-sensitive service, but fast decode alone does not establish prompt-processing latency. Their cache-capacity trade-offs still matter when interactive sessions grow long.

For high-concurrency batch inference and high-QPS pipelines, aggregate throughput shifts attention toward utilization across both phases. Disaggregated configurations address that mismatch by assigning prefill and decode separately. Long-document processing introduces a different constraint. There, cache format, residency, and low-cost offload set how much context capacity can grow without costs rising at the same rate. FP8 or FP4 KV storage and NVMe-backed eviction address that problem more directly than decode bandwidth alone.

For high-volume classification and RAG pipelines, GPU providers such as Together, Inference.net, and Fireworks fit the cost-focused path through FP8 and aggressive batching. The trade-off is latency variability under load. Hard sovereignty requirements steer the choice toward the European providers, because meeting those rules takes priority over how well the infrastructure performs. Across these cases, the most diagnostic provider question is concrete. What format holds the KV cache, where does it reside, how is it reused or evicted, and how quickly can it move when the workload needs it?