Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS
Microsoft's three-layer routing blueprint: how to scale agent traffic on Kubernetes without picking the wrong model for the job.
Why it matters
A reference architecture for production agent deployment on AKS addresses a real gap in the operationalization toolchain—model selection, call management, and replica placement—that enterprises moving agents from pilot to scale need to solve.
The key facts
8 to knowMicrosoft released a reference architecture for agent traffic routing on Azure Kubernetes Service (AKS)
Architecture breaks routing into three layers: model selection, call management, and GPU replica assignment
Targets enterprises deploying multi-model agent systems at scale
Three-layer architecture: model selection, call management, GPU replica routing
Deployed on Azure Kubernetes Service (AKS)
Reference architecture released (not a product announcement, but deployment guidance)
Targets agent traffic routing specifically
Published July 2026 — recent and timely to enterprises scaling agents
Go to the source
InfoQ AI/MLinfoq.com
Publisher excerpt: Microsoft has released a reference architecture for routing agent traffic on Azure Kubernetes Service. It breaks down the issue into three key choices: which model answers a call, how the call is managed, and which GPU replica handles it. By Claudio Masolo