Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model
26M parameters. That's all it takes to match 270M+ models on function calling—and it runs on your phone at 6000 tok/s.
Why it matters
Cactus open-sourced Needle, a drastically smaller tool-calling model built on a radical architectural insight: agentic capabilities don't require massive FFNs or reasoning capacity. By stripping MLPs and relying on cross-attention for retrieval-and-assembly, they've shown that function calling scales down dramatically—opening a new efficiency frontier for on-device AI agents.
The key facts
8 to knowNeedle: 26M parameters, beats FunctionGemma-270M on single-shot function calling
Inference speed: 6000 tok/s prefill, 1200 tok/s decode on consumer devices
Architecture: attention + gating only, no MLPs (Simple Attention Networks)
Training: 200B token pretraining (16 TPU v6e, 27 hours) + 2B token function-calling post-training (45 minutes)
Dataset: 15 tool categories (timers, messaging, navigation, smart home) synthesized via Gemini
Tested on 5 baselines: FunctionGemma-270M, Qwen-0.6B, Granite-350M, LFM2.5-350M
Finding generalizes: no-FFN architecture applies to RAG and any task with external structured knowledge
Open source: MIT licensed, weights on Hugging Face, part of Cactus inference engine
Go to the source
Hacker Newsgithub.com
Publisher excerpt: Hey HN, Henry here from Cactus. We open-sourced Needle, a 26M parameter function-calling (tool use) model. It runs at 6000 tok/s prefill and 1200 tok/s decode on consumer devices. We were always frustrated by the little effort made towards building agentic models that run on budget phones, so we…