Top
Best
New

Posted by get-inscribe 7 hours ago

Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor(get-inscribe.com)
409 points | 172 comments
satvikpendem 7 hours ago|
Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe.

However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.

jiehong 6 hours ago||
Also, this test is English-only, while a strong point of other models is to understand different languages without first having to say which one (so you don't need 3 different keyboard shortcuts if you wanna dictate in 3 languages day-to-day)
frereubu 5 hours ago|||
Reminds me of the time my neighbours must have wondered if I was having some kind of a breakdown when trying out really basic MacOS voice recognition back in the early 2000s. There was a keyboard shortcut and you could say something like "phone number for firstname lastname" and it would theoretically show you that phone number. Thing is it didn't seem to like a British accent, so I spent a good hour trying out different accents, rotating through various US accents, Australian, South African, Canadian and so on. It seemed to respond best to some kind of a melange of Californian / Australian.
PyWoody 4 hours ago|||
Scottish Elevator - Voice Recognition - https://www.youtube.com/watch?v=NMS2VnDveP8
frereubu 3 hours ago|||
Not too far off what happened, although thankfully I wasn't actually trying to do anything other than test it. Going to take the opportunity afforded by Scottish TV comedy here, and make a very tenuous link to intercultural exchange so I can post my favourite Rab C Nesbitt scene, hands across the sea indeed: https://www.youtube.com/watch?v=uKxPH_QH940
ntcho 2 hours ago|||
Thanks for this gem, had a good laugh
thebruce87m 3 hours ago|||
Interesting - I don’t think I’ve ever seen anyone from the UK refer to talking in a “British accent” before since we are normally aware of the wild regional variations.
frereubu 3 hours ago||
Fair point! I think it's a tic from being English and having lived in Scotland for quite a while so I autocorrect "English" to "British", but I've over-corrected here. (Also perhaps something to do with "British English").
arjie 5 hours ago||||
Does anyone have any experience with Mandarin STT? What's a good model for this? The use-case I have is subtitling of Mandarin speech.
tapland 1 hour ago|||
I use Systran/faster-whisper-medium for real-time subtitling, but you need to get used to the context it's used it and the weirdness it translates into. Parakeet has great mandarin>CN text, but running that + a translation model has been tricky and I never got it fast.
arjie 54 minutes ago||
Thank you for this too. I am running in an offline scope so I don't need speed just quality. I'm willing to do it overnight as well if required.
satvikpendem 2 hours ago|||
MOSS-Transcribe-Diarize [0] is by a Chinese team so apparently it's quite good. Try it out here [1].

[0] https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize

[1] https://huggingface.co/spaces/OpenMOSS-Team/MOSS-transcribe-...

arjie 2 hours ago||
Thank you very much.
verelo 6 hours ago|||
As an Australian, Apples voice models have always sucked. I've tried using stt (again) more recently and its improved, but i'm so tired of having to Americanize my voice to get it to figure out what the hell i'm saying.
jermaustin1 6 hours ago|||
As a Texan first, American second, I sympathize with this statement. Siri can't understand me probably 25% of the time. I use STT for iMessage while in the car, and half the time it will take 3+ times to either get it right or me give up, and hope to remember to text them by hand when I next stop.
danabrams 6 hours ago||||
It also struggles with my NYC-area accent, which is only medium thick.
MisterTea 3 hours ago||
The Two Yoots problem. Do you use d's in place of t's such as dees/dems/dose/dere? I have a heavy queens accent so you'll hear me say things like "deres tree uh dem ova dere."
llbbdd 2 hours ago||
Surprised to read this as a Queens thing, this sounds perfectly at home in the Midwest.
louthy 5 hours ago|||
As a Brit, I concur.
verelo 5 hours ago||
That checks out.
ChadNauseam 6 hours ago|||
> there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia

Is parakeet state of the art? It always transcribes speech fragments for me, like if I stutter and say "m-m-m-map" parakeet will dutifully transcribe "m m m map". Which I guess could be a good thing or a bad thing depending on what you want. Whisper does not do that however.

I do like cohere transcribe a lot.

robgough 3 hours ago|||
I think that's parakeet doing its job there. That is a closer reflection of what you've actually said. The trick is then throwing that output through some additional deterministic and non-deterministic steps to tidy it up however you prefer. It's exactly what I do with my free and open source dictation app (dictator.robgough.net) for Mac+iOS. And of course, everything stays entirely on-device. Gemma E4B is really wonderful for that second step, it's great at language – but takes up 6-7GB RAM.
dinfinity 4 hours ago||||
I use Parakeet V3 via this tool and it is actually quite reliable for me (in English): https://github.com/cjpais/Handy
EsotericSoft 4 hours ago||
If you are using Parakeet for English only then you should be using V2. V3 is for several languages and is worse at English only.
parentheses 5 hours ago||||
Agree on this point. Recent anecdotal testing I did found Whisper is still better than Parakeet.
obmelvin 5 hours ago||||
Parakeet is certainly faster on my machine (m3 max), but I can't stand using it over Whisper for dictating my prompts. It makes a lot more mistakes, possibly because (like you mentioned) large portions of the speech will pause / stutter as I think about what to include.

With whisper v3 turbo, I can almost always live with the few mistakes because the overall stream-of-thought word-salad I provide is still clear at a high level. The bits and pieces of context seem to help, that I might leave out if typing and focused more on traditional conciseness / clean writing. With parakeet, I needed to do frequent editing even for shorter bits of speech.

I realize some applications prioritize the latency.

solenoid0937 4 hours ago||
It sounds like post processing should be the job of an LLM. I would like the voice model to be faithful to what was said and then that output can be smoothed over or postprocessed as needed for the use case
obmelvin 4 hours ago||
To be clear, I'm talking about high word error rate with parakeet vs whisper, not post processing and cleaning up my speech. Re: being faithful to what was said, one small example, Whisper will often put ellipses when I pause.
satvikpendem 2 hours ago|||
No, there are better open weight models: https://artificialanalysis.ai/speech-to-text/non-streaming

Apparently MOSS-Transcribe-Diarize is quite good too as it released only a few days ago.

ComputerGuru 1 hour ago|||
Whisper v3 is still the best (by far) when it comes to poor quality input (say background audio from a security camera), though remains more susceptible to hallucination so it's a bit of a tradeoff.
athnak 6 hours ago|||
Apple's own Voice Memos app already does automatic transcription since macOS 15 / iOS 18.
al_borland 5 hours ago||
Speech-to-text is also already built into the keyboard as well, so it can be used in any app where a user would type.
hectdev 5 hours ago||
From my experience, Speech-to-text falls way short of Wispr flow and I would assume the ones that are said to be better than that. It lacks context awareness and formatting
orbital-decay 5 hours ago|||
Of these only Parakeet is <1B, it looks better than Apple's model, however it's not builtin. I wonder how it compares on latency and efficiency.
hendersoon 5 hours ago||
Parakeet is incredibly fast and accurate even on CPU, and it supports streaming now also in TDT3.
parl_match 5 hours ago||
Apple likely needed a model that ran on their NPU natively.

- parakeet usually runs on Bfloat16. NPU doesn't support that

- CPU is not as fast as the NPU for these ops on A-series, and even on modern CPUs, there's a latency delay

- Parakeet latency is fine but "fine" may not be good enough for Apple's UX team.

- CPU increases power consumption over dedicated float blocks

So I would say that Parakeet was a non-option for Apple to ship, although it should be in the benchmarks anyways!

foobarqux 5 hours ago||
Fluidaudio implements Parakeet on ANE. I'd like to know how SpeechAnalyzer compares in speed.

https://github.com/FluidInference/FluidAudio

hamza_q_ 21 minutes ago|||
Recently contributed a patch to FluidAudio that sped up Parakeet V2 and V3 to 320x and 282x faster than real time, respectively:

https://github.com/FluidInference/FluidAudio/pull/507

That means one hour of audio transcribed in 11.25 and 12.75 seconds.

The Inscribe post doesn't give a speed factor for SpeechAnalyzer. However, this Argmax blog post reports 70:

https://www.argmaxinc.com/blog/apple-and-argmax

Based on that, FluidAudio is ~4.6x and ~4.0x faster.

foobarqux 4 hours ago|||
Just tried test using yap on a single ~1hr mp3: yap/Speechanalyzer is about 50% slower than fluidaudio on M1. yap interface is nicer though.

https://github.com/finnvoor/yap

saturn8601 5 hours ago|||
I hope they replace their awful voice to text on their keyboard. I can't stand that terrible bit of software.
FuckButtons 4 hours ago|||
Yeah, apple will be optimizing a model to work on ANE and then turn it into a native app. My only hope is that it has a reasonable api so that I can use that as a generic input source across iOS / macOS that’s equivalent to the ubiquity of the keyboard.
swiftcoder 3 hours ago||
Presumably the existing transcribe button on the keyboard will route through this on iOS 26?
Adrig 5 hours ago|||
> RIP to a lot of the paid apps that simply wrap Whisper

I started using a few open source apps for transcription and eventually subscribed to a paid one...

On paper, it's not hard to compete, but for this use case, a few rough edges make it really frustrating to use. Like a keyboard that sometimes doubles the letter "e"

Automatic dictionary, seamless language switch, no issues with accents, etc... Putting the effort in the last mile makes a world of difference.

If anyone has better options, I'm willing to have a look. The best open source solution I found was Handy, and I currently use Wispr Flow

robgough 3 hours ago||
I built my own because I was frustrated with a lot of the free options. Largely because a lot of them had an upsell to be able to do the secondary post-processing step with an LLM. And it wouldn't pick up things like emojis properly or say numbers. Because of that, I left quite a lot of options in there for customising and adding additional steps, etc. Feel free to take a look: dictator.robgough.net

My initial Mac version actually had three additional steps that you could toggle, obviously at the cost of some speed. That is what the website talks about, although nowadays for my own use I've reduced that to just one step and found that it's pretty great. I've got a new version in test to tidy that up, but still lets you define as many steps as you want.

z2 4 hours ago|||
I don’t know how Apple divides computation between the GPU and the Neural Engine, but one major benefit, especially for real-time transcription on laptops, is the improved power and thermal efficiency. I noticed better accuracy after switching my app to SpeechAnalyzer, and I suspect part of that improvement for me came from the microphone no longer having to compete with jet-engine fan noise.
wahnfrieden 6 hours ago|||
For multilingual and noisy audio the best right now is MOSS-Transcribe-Diarize which was released just a few days ago

Superwhisper does a lot more than just provide a whisper/parakeet UI so I’m not sure Apple will destroy them so easily

satvikpendem 6 hours ago|||
Thanks, was looking at a better diarization model.

Even for those sorts of apps, MacParakeet which I've been using is FOSS so no payment needed. In reality these days with AI the ability to spin up a free and/or OSS competitor falls to zero.

wahnfrieden 6 hours ago||
I’m not even using it for diarisation just transcription and it’s amazing. It also doesn’t need a VAD

A new VAD I found though is FireRedVAD which has better benchmark results than TEN and Silero by far

satvikpendem 2 hours ago||
Is MOSS a streaming model or only for offline? For that VAD how are you integrating it into a model like Whisper etc?
wahnfrieden 2 hours ago||
I'm using it offline. But it's much faster than realtime so it should be usable for streaming. I just asked Codex / Sol to integrate FireRedVAD with Whisper...
techsystems 5 hours ago||||
Interesting! And what would you say are MTD top competitors?
wahnfrieden 3 hours ago||
I'm only dealing with Japanese audio so for me it's Anime-Whisper, a Japanese-specific fine-tuning of Qwen3 ASR, and Apple SpeechTranscriber.
foobarqux 5 hours ago|||
16GB! (edit: this is wrong, I was looking at TTS, the transcribe model is 1.7GB). Compared to Parakeet 2.3GB (but no diarization).

Also doesn't seem to be tailored to Apple hardware (i.e. no MLX or ANE variant/implementation)

wahnfrieden 3 hours ago||
I am running the MLX fork https://huggingface.co/vanch007/mlx-MOSS-Transcribe-Diarize

Generally labs don't release MLX or ANE versions and we must rely on finding someone who's converted it

Parakeet is not multilingual so not directly comparable

Where do you see 16GB? MOSS is smaller than Parakeet at 1.82GB

foobarqux 3 hours ago||
Thanks, I didn't see vanch007 version at first (only ~30 downloads), I usually look at mlx-community. For the size I was looking at the wrong model (TTS not transcribe-diarize), thanks for the corrections.
enkonta 5 hours ago|||
I’m not sure I agree. There may be better models, but the comparison is still useful so long as whisper is so widely used.
jasondigitized 3 hours ago|||
I am curious, what are the use cases people are using voice transcription for?
jlund-molfese 3 hours ago|||
I find voice memos really annoying, so I’ve been dropping them into a Whisper frontend for a while. Something built into the OS would be nicer though.
Computer0 2 hours ago||||
The canadian government will provide lots of historical data for curious citizens, many of which are recordings of interviews from decades and decades ago. For a book project this allows me to make a hours of audio searchable through a GUI application I have developed that has a voxtral backend.
iAMkenough 2 hours ago|||
Turning dialogue audio into text for searchability and accessibility (particularly for deaf users)
elAhmo 4 hours ago|||
I use handy.computer and it is pretty much everything I want from a transcribing app.
permalac 5 hours ago|||
Hey. Yes. I did vive code one as an exercise yo learn how to publish to apple store.

Listen and transcribe felt like the easiest thing to do.

Distavo.com

The source is open for anyone to use, and the builds are in github.

I found quite interesting that claude didn't help too much on how to publish to SetApp until Fable.

bellowsgulch 5 hours ago|||
> However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.

What's insane to me is that you have all of these low-quality me-too apps, and literally no one could bother to read the damn Human Interface Guidelines or follow iOS design conventions.

Doing so is literally LESS WORK than trying to make your own custom awful iOS UI.

ChrisMarshallNY 5 hours ago||
Not if your app is a Web wrapper, which so many of these are.

If you use SwiftUI (the native recommendation by Apple), it severely penalizes you, if you want to paint outside the lines (which is a big reason that I don't use SwiftUI for shipping apps). It's insanely easy to write a native app that is 100% in line with HIG.

BeetleB 4 hours ago|||
How many of the Whisper competitors will work at a reasonable speed using only CPU (on Linux, not Apple)?

(Genuine question - I'm a happy Whisper user but am always looking for improvements).

llm_nerd 4 hours ago|||
This particular product used Whisper, so that was obviously the right model to compare it against. Further this is explicitly on device, and Nemotron 3.5, as one example, is 2.5GB for the model.

And if someone were broadly comparing all on-device models (instead of just looking at how this new on-device ones compares to what a specific product uses), Nemotron 3.5's WER are actually a bit higher than what they report for SpeechAnalyzer, for both tests.

trencedamp 6 hours ago||
Came here to post this. I use handy on my own machine and it's perfect with parakeet. If I switch to whisper it makes lots of mistakes
ashivkum 7 hours ago||
Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.
seviu 7 hours ago||
Been using it for a podcast app I have been developing for half a year lol (I hope I publish it by version 27) and I can confirm it’s real fast.

Splitting the audio in multiple segments and firing it up without hitting the maximum limit of concurrent decoding streams makes it blazing fast. Fair enough you loose the cut, but it’s good enough for just podcast. In one minute it chews through one hour of audio. This on an iPhone 17 Pro.

rokkamokka 4 hours ago|||
You could perhaps run over the segment splitting points (plus a few seconds back and forward) in a second batch then merge the results in the end so you don't miss anything.
satvikpendem 7 hours ago|||
What's different about your podcast app?
seviu 6 hours ago||
Nothing really, except that I get to play with SpeechAnalyzer APIs, foundation models, translations. It’s basically my playground where to try all things. Been listening a lot of Chinese podcasts lately, transcribed and translated by local models.

Edit: all that said, the app is irrelevant. What I want to say is that live transcripts on iOS using Apples frameworks works very well. Only thing I miss is diarization support.

Chu4eeno 7 hours ago||
If it was faster but worse, maybe compare it to a smaller whisper model?
generalizations 7 hours ago||
I imagine because quality of transcription is what matters.
mchusma 5 hours ago||
I will plug Willow for mac recording. IMO it's basically to me a "better than perfect transcription" as it cleans things up and is almost instant. I liked Superwhisper but switched to Willow as it was a big difference.

Its so good that I'm not sure that it's possible to get any better. Speech to text seems like basically a solved problem, if not now then definitely in 5 years. I don't know if any of these speech to text businesses will work in the long run, but for consumers they are great. My guess is the 2030 version of Apple's SpeechAnalyzer will be so good that nobody will need to use 3rd party software.

tcoff91 13 minutes ago|
What isn't solved is domain-specific jargon with these tools. When i talk to my coding agent, i want to be able to speak the names of symbols and files and have it be aware of that stuff, like having LSP integration.

If I say 'useSuspenseQuery' I want it to come out as useSuspenseQuery not 'use suspense query'. Even if I had to say 'symbol useSuspenseQuery' to give a hint that i'm referencing a symbol, that would be fine.

akurilin 1 hour ago||
I took a swing at bringing this into Handy.computer if anybody's interested: https://github.com/cjpais/Handy/discussions/1031 . Looks like there has been past demand for someone to implement it, but no proposed PRs. This article was inspiring.
thih9 54 minutes ago||
> Apple's new SpeechAnalyzer is the most accurate on-device speech engine we tested. It beat every Whisper model we ship, including Whisper Small, on both the clean and the noisy half of LibriSpeech, while running roughly three times faster than Small.
summarity 6 hours ago||
Vs Voxtral would be a better comparison. No other model, open or closed, has been able to hit such a low AER (Acronym Error Rate ;)) for my meeting transcripts. Seems to understand/infer all the technobabble I use at work. Never have to edit anything. Whisper was catastrophically bad.
pants2 5 hours ago|
I typically disable autocorrect on Apple products because of this, cautiously optimistic about their improved speech models, but definitely worried that it's going to 'correct' technical jargon to more common words.
MBCook 7 hours ago||
Impressive. Apple said they improved the models in 27 didn’t they? It would be interesting to see the numbers the beta turns in.
modeless 7 hours ago||
Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?
satvikpendem 6 hours ago||
There are many [0], you can search and filter by streaming and open weight only as well.

Looks like Voxtral and Nvidia's Nemotron are best.

[0] https://artificialanalysis.ai/speech-to-text/non-streaming

Chu4eeno 7 hours ago||
There's tons, Parakeet was the last I remember seeing which seemed to gain traction (independent lightweight implementations etc).
garblegarble 6 hours ago|||
I have tried everything (that will run on a 12GB RTX 4070) and I have yet to find anything with better accuracy than Whisper V2 Large for my dataset (discord audio from TTRPG sessions, isolated per-speaker, mostly non-American accents)
Ey7NFZ3P0nzAe 2 hours ago|||
Voxtral to me what better
xd1936 6 hours ago||||
Same, for my English-only podcast
modeless 5 hours ago|||
Not v3?
satvikpendem 6 hours ago||||
Nvidia's Nemotron subsumes their older Parakeet model now even for real time streaming.
daemonologist 7 hours ago||||
Parakeet is way faster (on Nvidia hardware) but not quite as accurate in my experience.
meatmanek 6 hours ago||
It's also super fast on CPU.
behnamoh 7 hours ago|||
Parakeet isn't as good as whisper large.
wingworks 5 minutes ago||
I wish Apple would update there built in TTS model (on Mac). In this day and age, it's quality is terrible.
wgm 4 hours ago|
Is this the new dictation engine that I'm not allowed to run on my 1-YEAR-OLD IPHONE 17 because it's not Pro?
monster_truck 4 hours ago|
Why did you buy the cheap one, that's your own fault
wgm 4 hours ago|||
At the time, there was very little advantage to buying the Pro model. Ironically, it's the first time I've ever bought the base model.
monster_truck 2 hours ago||
Serves you right
satvikpendem 2 hours ago|||
I hope this is sarcasm.
More comments...