Presentation: Producing the World's Cheapest Tokens: A How-to Guide
Order-of-magnitude cost cuts in LLM inference: the hardware-software trade-offs that reshape cloud economics.
Why it matters
As token costs become a primary cost lever for AI deployments, practitioners need the architectural playbook for non-real-time inference — hardware selection, runtime optimization, and queuing strategies that can cut costs by 10x or more.
The key facts
10 to knowFocus on batch/non-real-time inference workloads (highest leverage for cost reduction)
Hardware trade-offs: balancing GPU/TPU choice, memory, and throughput
Inference runtime optimization and speculative decoding as cost multipliers
Queue reordering and scheduling strategies for utilization
Targets software architects and engineering leaders (buyer/decision-maker audience)
Focus: low-cost inference architectures for non-real-time, high-volume workloads
Key levers: hardware selection, inference runtimes, speculative decoding, queue reordering
Target: order-of-magnitude cost reductions through architectural trade-offs
Audience: software architects and engineering leaders optimizing token economics
Speaker: Meryem Arik (infrastructure/inference optimization expertise)
Go to the source
InfoQ AI/MLinfoq.com
Publisher excerpt: Meryem Arik discusses strategies for designing low-cost LLM inference architectures for high-volume, non-real-time workloads. She explains how software architects and engineering leaders can achieve order-of-magnitude cost reductions by making critical trade-offs across hardware, inference…