Posted by HenryNdubuaku 5 days ago
We submitted Needle 2 here a few weeks ago, and the feedback in the discussion thread was incredibly valuable, thanks! Thanks to all that feedback, we’ve been able to move quickly to release Needle 3 and I'd love to hear what you think again.
The key features:
1) Automation (tool calls & structured JSON output): Needle still doesn't chat by design, its quite challenging to pack general capacity into such small models, so we focus on tool calls and structured JSON. If no tool you declared fits the request, you get an empty list back (note for when playing with the demo).
2) Intelligence Laddering: Every layer (2 to 20) is a deployable subnetwork, so one set of weights, 25 to 121 million parameters at 2-bit, shipping as 8-29MB binaries. On a Raspberry Pi 5 it decodes at up to 4k tokens/sec and prefills at up to 10k.
3) Monarch Hadamard MLP: replaces the dense FFN with three learnable Walsh-Hadamard-initialized Kronecker (Monarch) factor pairs interleaved with per-channel diagonal scales, fixed permutations, a SiLU nonlinearity, and a rank-8 input-conditioned gate, so each token gets a fully mixed nonlinear transform of its d_model channels at O(d√d) parameters and compute instead of the O(d²) a dense 4x-expansion MLP would cost.
4) Performance: On Mobile Actions (phone commands, scored on the exact call) the 20-layer model gets 86.0 through the shipped 2-bit binary; LFM2.5 1.2B is at 82.4, Qwen3.5 0.8B at 76.0, Apple's on-device model at 57.6, all at f16. More results on the link, we do not win everywhere ofc.
5) Multilingual: Needle 3 now supports English, French, Spanish, German, Dutch, Italian, Polish, with more languages coming.
6) Finetuning: You can achieve DeepSeek v4 Flash grade performance on a narrow task with just 4L, stress on "narrow task", we found that production users often prefer tuning before production.
7) Triggers: Grounding is a common challenge for tool call, at least for Needle 2, so we added support case-insensitive regular expressions matched against each request to gate false negatives.
8) Confidence: Every response also carries a calibrated confidence score, the minimum of a judgement on the finished call and its decode probability. Act above your threshold, show the call and ask below it, or escalate to a bigger model.
9) Supported Platforms: macOS, Linux on x86-64, ARM64, ARMv7, RISC-V and MIPS32, Windows x64 and ARM, Android, iOS, watchOS, tvOS, the browser as WebAssembly, and a WASI component.
Thanks for reading and as always, thoughts appreciated!
"I need a wee" -> tries to play music because "wee" is a genre
"I need a wee wee" -> starts the vaccuum in the bathroom
"I'm going to the toilet" -> says it'll turn on the toilet, and I'm not totally sure what that entails.
"I'm going to the toilet and can't see" -> reasons that lights should be on in the bathroom, then chooses again to turn on the toilet.
"I'm going to the toilet and can't see where I'm going" -> reasoning is "'going to the toilet' -> control_device with device 'coffee maker' (toilet implies coffee maker)"
"I'm going to the toilet and can't see where I'm going because it is too dark" -> "'dark' -> direction 'dark'; adjust_lights with brightness 100 for darker light"" and chooses to turn the lights in the living room to "dark" which fails.
At this point the vacuum is in a dark bathroom, the living room is 100% brightness and playing "wee". At least there's coffee.
> 'sleepy time' means sleeping → start_vacuum with room 'bedroom' to start cleaning
The "DeepSeek 4 Flash grade" claim seems far fetched.
Edit - I’m struggling to get anything useful. Reasoning is often utter nonsense and the actions are very often very wrong. To the point of seemingly needing very precise sentences to work at which point you may as well do regexes. Very simple things like clean one room then another with the vac fails.
Or maybe it just has a weird thermostat down bias? "make it hot" also had it turn it down (specifically it went from 20->18, or at least tried to, the UI still showed 20), with high confidence. Also might have a bit of a Celsius vs Fahrenheit confusion. Neat concept, but I might not want to let it control the oven at the moment.
The laptop demo worked better until I tried to open the mail app. "Check mail" kept opening the browser with an error, and "check email" makes a note with the text "email", "open email" goes to "https://api.email.com/v1/email" in my real browser, but "open mail" does work.
And I presume the "reasoning" isn't very trustworthy? In the car I got "'turn it up' means lower volume -> set_volume with lower value." For the house, reasoning would correctly say that I wanted the alarm off, but it didn't actually do it.
I'm a big fan of OpenStreetMap, and I enjoy editing it from my computer. From my phone, I find it quite tedious trying to make sure I type in the phone number exactly correctly and double-check it, or find and select the right field from the large list of fields available in Upredor.
Generally, how it works is I see a restaurant, and there's a sign. I know that it says, "Cash only. Here's the phone number. Here's the opening hours." What would be really cool is if I could just speak to the phone and say, "Hey, here's the information about this place." It would automatically use your location to detect what places are nearby and maybe even detect which place you're talking about, and then tell you, "Okay, here are the changes I think you're proposing to make, or these things you stated are the ones that would create a diff." This would be limited to just perhaps the 20 most common keys in some predefined set of values for most of them. Like cuisine=x should just match to the most common not make up new ones.
Of course, this is something a large language model could do, but having it run on device would be a lot nicer and cheaper.
For "which place", query nearby POIs from location in the app and pass the candidate names as an enum field, so Needle picks rather than guesses.
Two caveats: it's text-in, so you need on-device STT first, and opening_hours syntax is the risky bit, so either put the format in the description or capture the raw hours and normalise in code.
What's more important than the resource requirements is to highlight what the model simply cannot even attempt to do that general LLMs do decently well.
In other words, tell me the anti use case clearly so that I don't have to find out myself.
All of the other tasks a general-purpose LLM can do (write me a poem about pizza, rewrite this code in rust, tell me about the causes of the war of the roses) are unsupported.
The only use case this supports is converting unstructured text into structured json calls, and doing that quickly in a low memory environment.
Model Correct tool shape Exact arguments
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━
FunctionGemma fine-tune, BF16 209/230 (90.9%) 196/230 (85.2%)
─────────────────────────────── ──────────────────── ─────────────────
Needle 3 fine-tuned W4A8 74/230 (32.2%) 47/230 (20.4%)
─────────────────────────────── ──────────────────── ─────────────────
Needle 2 fine-tuned W4 59/230 (25.7%) 43/230 (18.7%)Jev has put the cat amongst the pigeons so suddenly everyone is looking at classifiers and encoder only models again.
My ideal model would be a general purpose LLM API that can answer classification questions and as it does so distils to an encoder only model so that the more classifications I do the cheaper it gets (i.e. the more it offloads to the classifier). If anyone ever wants to do this as a service do let me know, because it's just another piece of code to manage in each new project that needs classification.
Also a model that could do this internally would be nice :-)
For instance, you want to be able to handle any smart home commands people could issue, right? What are all of the smart home devices? What are all of the ways people might want to issue commands? Also, for things like Spotify, it’s not going to know what “The Beatles” are or “Led Zeppelin”. Artist and song names themselves are easily just as hard as all of the smart home devices combined.
The simple attention network stuff is cool, it makes sense to drop the MLP when it dominates the param count. But you’ll definitely lose some “world knowledge”. That’s probably ok though.
I have a test suite that tries like ~36 different scenarios, including things like starting multiple timers, saying "actually cancel that timer" and whether it knows to do that one you just created. Basic decision making on top of tool calling. I found so far that, for example, Qwen3.8 on my local machine does pretty poorly even relative to Gemma4 E4B (~9.6gb) and that the best price/performance outcome I've found so far with openrouter is actually GPT Luna, but obviously I'd love to get something that works as well running locally for privacy reasons.
Would love to try this out, I'll just need to tweak my benchmarker to use however this serves it.
For your question on tool calling, I think you will find that the model is pretty good at simpler tool calls and parallel ones, but can struggle with implied references and multistep reasoning. These are definitely things that can improve with task-specific finetuning but for some things you just have to have a model that is properly sized. That said, we are always trying to improve the model so that it can handle an ever larger set of queries
What it gets wrong:
- It copies numbers instead of converting them. "25 minute timer" becomes duration_seconds: 25, and "twelve minutes" becomes 120. The first one comes with 100% confidence.
- It picks the wrong action. "take the paper towels off the list" became an add. "remind me in 20 minutes" became a timer. "add five minutes to the pasta timer" became a new timer plus a cancel.
- It never declined anything with our full tool set. Background chatter became note_save "blue one" at 0.99 confidence. "play some jazz" became a screen card, and "wake me up at 6 30" became a 630-second timer.
- It can't use household context. Notes, timer names and reminder IDs have no place in its input. Passing them anyway made results worse (5 of 27 single-turn requests right, versus 8 of 27 without), so the backend now leaves them out.
- Follow-ups mostly broke. "take off the last one" removed the whole list.
Wow, I was just researching W-H in transformers. Did yours seem to work? In my experiments swapping various components for W-H-like transforms caused extreme quality degradation.
UPD. according to the comments here, this model simply does not work at all, so I guess the answer is NO