Needle: We Distilled Gemini Tool Calling Into a 26M Model
Cactus Compute open-sourced Needle, a 26M-parameter tool-calling model that reaches 6,000 tok/s prefill and 1,200 tok/s decode on consumer devices, using an attention-and-gating architecture with no MLPs.
Why it matters: Cactus Compute distilled Gemini's tool calling into a 26M-parameter model and reports 6,000 tokens per second prefill on consumer devices, a lightweight option for on-device tool use.