Latency

Definition

The time between sending a request to an AI model and receiving the first token of the response. Low latency is critical for real-time applications like coding assistants, voice agents, and live customer support.

Why it matters

Latency determines user experience. Research consistently shows that response times above 2 seconds cause significant user drop-off, and above 5 seconds, most users abandon the interaction. For real-time applications like voice agents, even 500ms of latency feels unnatural. Latency optimization involves every layer of the stack: model architecture, hardware selection, geographic placement, inference engine, and network. The trade-off between latency and cost is a core architectural decision: smaller models are faster but less capable; larger models are more capable but slower. Streaming (returning tokens as they are generated) mitigates perceived latency but not actual time-to-completion.

In practice

Groq's LPU achieves time-to-first-token of under 100ms on Llama models, roughly 10x faster than GPU-based inference. Edge deployment brings models closer to users, reducing network latency from 100-200ms to under 10ms. Voice AI companies like ElevenLabs and Hume target end-to-end latency under 500ms for natural conversation. Anthropic and OpenAI offer streaming responses that return tokens as generated, reducing perceived latency to under 200ms even for complex queries. In enterprise benchmarks, the difference between 200ms and 2000ms latency correlates with 3-5x differences in user satisfaction and feature adoption.