Posted by HenryNdubuaku 5 days ago
We submitted Needle 2 here a few weeks ago, and the feedback in the discussion thread was incredibly valuable, thanks! Thanks to all that feedback, we’ve been able to move quickly to release Needle 3 and I'd love to hear what you think again.
The key features:
1) Automation (tool calls & structured JSON output): Needle still doesn't chat by design, its quite challenging to pack general capacity into such small models, so we focus on tool calls and structured JSON. If no tool you declared fits the request, you get an empty list back (note for when playing with the demo).
2) Intelligence Laddering: Every layer (2 to 20) is a deployable subnetwork, so one set of weights, 25 to 121 million parameters at 2-bit, shipping as 8-29MB binaries. On a Raspberry Pi 5 it decodes at up to 4k tokens/sec and prefills at up to 10k.
3) Monarch Hadamard MLP: replaces the dense FFN with three learnable Walsh-Hadamard-initialized Kronecker (Monarch) factor pairs interleaved with per-channel diagonal scales, fixed permutations, a SiLU nonlinearity, and a rank-8 input-conditioned gate, so each token gets a fully mixed nonlinear transform of its d_model channels at O(d√d) parameters and compute instead of the O(d²) a dense 4x-expansion MLP would cost.
4) Performance: On Mobile Actions (phone commands, scored on the exact call) the 20-layer model gets 86.0 through the shipped 2-bit binary; LFM2.5 1.2B is at 82.4, Qwen3.5 0.8B at 76.0, Apple's on-device model at 57.6, all at f16. More results on the link, we do not win everywhere ofc.
5) Multilingual: Needle 3 now supports English, French, Spanish, German, Dutch, Italian, Polish, with more languages coming.
6) Finetuning: You can achieve DeepSeek v4 Flash grade performance on a narrow task with just 4L, stress on "narrow task", we found that production users often prefer tuning before production.
7) Triggers: Grounding is a common challenge for tool call, at least for Needle 2, so we added support case-insensitive regular expressions matched against each request to gate false negatives.
8) Confidence: Every response also carries a calibrated confidence score, the minimum of a judgement on the finished call and its decode probability. Act above your threshold, show the call and ask below it, or escalate to a bigger model.
9) Supported Platforms: macOS, Linux on x86-64, ARM64, ARMv7, RISC-V and MIPS32, Windows x64 and ARM, Android, iOS, watchOS, tvOS, the browser as WebAssembly, and a WASI component.
Thanks for reading and as always, thoughts appreciated!
"My car crashed I need help"
{ "'crashed' implies need for music. 'play_music' with query 'car crashed' from user's words.", ..., "confidence": 1, }
> 'call 9 1 1' -> call_contact with name '9 1 1', no required params.
> 'call ambulance' -> call_contact with name 'ambulance'.
> 'call 911' -> call_contact with name '911' and no required params.
Only the last one actually used the emergency_sos function (even though the reasoning says it used call_contact). If I were to use needle in my application, how would I improve accuracy?
Esp paired with a small Whisper or Parakeet voice model, this enables some amazing real-world use cases in lower-power situations (car, marine, home, PLC, industrial automation). Combining with a solar/wind combo for your home could be incredibly interesting.
- "more light"
- "less light"
- "both doors should be locked"
- "if blinds are open, open back door"
Are there perhaps some industrial or agri use cases?
So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.
All of the demo setups (smart home, robot vacuum, watch, etc) could easily have access to a bigger model running on a more capable device either locally or via the internet.
And needle is one of the most promising model due to its original architecture (but we still need to finish building the actual eval dataset before making out final call).
But yes we are also considering bigger models, though we'll pick the smallest model of sufficient quality because not having to download a 600MB bag of weight is a feature in itself.
Thank you!
Made the robot vacuum clean the living room. Might be good to give it an idea of where items likely are?
Very cool though. I see a lot of potential.
It’s not replacing anyone, it’s not going to destroy our energy infrastructure because that was the only way to turn off a toaster that it wanted to turn off.
Nice work.