Sort providers by cost, latency, or throughput on AI Gateway
AI Gateway now sorts providers by cost, latency, or throughput in real time—no code changes needed.
Why it matters
Vercel's AI Gateway expands routing controls, letting builders optimize for cost, speed, or throughput dynamically. For teams managing multi-provider inference, this reduces waste and latency without engineering overhead.
The key facts
11 to knowThree new sort criteria: cost (lowest price first), TTFT/latency (milliseconds), TPS/throughput (tokens per second)
Ranking computed at request time—price changes and latency shifts flow through automatically
Compatible with Zero Data Retention (ZDR) and custom provider ordering
Routing metadata now shows which providers were considered, metric values, attempt order, and health status
Use case example: GPT OSS 120B has 5+ providers with noticeable cost/speed variation
Fallback to next provider only on unavailability of higher-ranked option
Three sort options: cost (lowest price first), TTFT (lowest latency), TPS (highest throughput)
Ranking computed at request time — price changes and latency shifts flow through automatically
Compatible with Zero Data Retention (ZDR) and explicit provider ordering
Routing metadata shows providers considered, metric values, attempt order, and health status
Example: GPT OSS 120B has 5+ providers with noticeable cost/speed variation
Go to the source
Vercel Blogvercel.com
Publisher excerpt: You can now sort the providers behind a model by cost, time to first token (TTFT), or throughput (TPS) in .AI Gateway The default provider order blends provider reliability, quality of model output, cost, and speed of response. You can now use for explicit control over ranking criteria.sort For…