Build realtime voice agents on AI Gateway

now supports audio/voice. You can add realtime voice, text to speech, and speech to text with the same calls you already use for text, image, and video, routed through AI Gateway alongside every other modality.AI Gateway Audio launches with models from and . Each call gets the same provider routing, observability, spend controls, and bring-your-own-key support you already use for your other models.OpenAIxAI These capabilities are in beta and available in 7.AI SDK Realtime, speech, and transcription model are supported on .AI SDK 7 Realtime turns your app into something a user can hold a conversation with. When they speak, the model responds right away. Because it replies in the moment instead of waiting for a full turn, users can interrupt and talk over it the way they would with a person. It fits voice assistants, customer support agents, hands-free tools, and anywhere a user would rather talk than type. What sets it apart from chaining models together is that a single realtime model hears audio and produces audio directly, instead of running a speech-to-text, then language model, then text-to-speech pipeline. In the browser, the hook manages the WebSocket connection, microphone capture, and audio playback.useRealtime The connection is authenticated with your AI Gateway credential, so you mint a short-lived token on the server and hand the browser only that token. Your API key never reaches the client. Add a route that mints the token: Then connect from a client component: The hook captures the microphone, streams the audio to the model through AI Gateway, and plays back the spoken reply. Outside the browser, you can drive the session over a WebSocket yourself with , , and . See the for that path.getWebSocketConfigserializeClientEventparseServerEventrealtime reference A realtime session works differently from a normal model call: Generate spoken audio from text with . Pass a voice and an output format, then write the result to a file:generateSpeech Transcribe r...