Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
Apple just published a method to slash LLM serving costs by optimizing KV cache across model depth—a lever competitors haven't fully exploited.
Why it matters
Apple Research proposes Stochastic KV Routing, a novel optimization technique that reduces KV cache memory footprint by leveraging depth-wise sharing rather than temporal compression. This directly impacts inference cost economics and serving efficiency—critical for competitive LLM deployment.
The key facts
10 to knowApple Research paper on KV cache optimization
Focus on depth-wise (layer-level) cache sharing as alternative to temporal compression/eviction
Targets LLM serving cost reduction via memory footprint
Autoregressive generation efficiency
Published May 2026 on Apple ML Research
Focus: Key-Value cache memory reduction via depth-wise sharing
Problem addressed: KV cache memory footprint as major serving cost driver
Approach: Orthogonal optimization along depth dimension vs. prior temporal axis work
Source: Apple Machine Learning Research (credible research arm)
Context: Relevant to LLM inference efficiency and production cost optimization
Go to the source
Apple Machine Learningmachinelearning.apple.com
Publisher excerpt: Serving transformer language models with high throughput requires caching Key-Values (KVs) to avoid redundant computation during autoregressive generation. The memory footprint of KV caching is significant and heavily impacts serving costs. This work proposes to lessen these memory requirements.…