The DropJuly 9, 2026via InfoQ AI/ML
AlloyDB Ships Proxy Models That Replace LLM Calls with Local Inference Inside the Database
Why it matters
Google is collapsing latency and cost for AI-powered database queries by replacing external LLM calls with locally-trained proxy models. This shifts the economics of AI apps from cloud-dependent inference to on-premise database-native execution.
Key signals
- AlloyDB AI functions reached GA with proxy model architecture
- 2,400x throughput improvement via smart batching
- 100,000 rows per second in preview (internal testing)
- Proxy models trained from LLM outputs, run inference at database speed
- Eliminates external LLM API calls for query execution
- Published July 9, 2026
The hook
2,400x. That's the throughput gain Google just shipped inside AlloyDB—no external LLM calls required.
Google shipped AlloyDB AI functions GA with a proxy model architecture that trains a lightweight local model from LLM outputs, then runs queries at database speed without external calls. Smart batching delivers 2,400x throughput improvement. The proxy model reaches 100,000 rows per second in preview, but benchmark numbers apply only to ai.if in internal testing.
By Steef-Jan Wiggers