Introducing Amazon SageMaker HyperPod Inference Gateway
82% faster first-token latency. Amazon just shipped a Kubernetes routing layer that makes inference servers smarter without touching your models.
Why it matters
SageMaker HyperPod Inference Gateway is a production inference optimization tool that reduces latency through GPU-aware request routing on EKS — actionable for teams running inference at scale who want plug-and-play performance gains.
The key facts
9 to knowKubernetes-native routing add-on for Amazon EKS
GPU-aware request routing using real-time GPU signals
Up to 82% reduction in first-token latency
No changes required to model servers or client applications
Part of Amazon SageMaker HyperPod family
Amazon SageMaker HyperPod Inference Gateway launched
Kubernetes-native, GPU-aware routing add-on for Amazon EKS
Real-time GPU signal-based request routing
Published September 18, 2026
Go to the source
AWS Machine Learning Blogaws.amazon.com
Publisher excerpt: Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cutting first-token latency by up to 82% with no changes to your model servers or client applications.