Top
Best
New

Posted by amrrs 11 hours ago

We turned EmbeddingGemma 2 into a speech-to-text model(eg2.haku.sh)
371 points | 82 commentspage 3
martianvoid 11 hours ago|
Pretty cool application of an embedding model
chronogram 11 hours ago||
What about the quality of the transcription?
walrus01 11 hours ago||
They say its word error rate is about 5%, so not great. Interesting as a proof of concept. The model this is based on is REALLY tiny.
koolba 9 hours ago||
Too bad the errors for things like this are not truly independent.

Otherwise could run two passes to get down to .25% or three passes to .0125%. Or four passes to .000625%!

That’d be .02 seconds of error per hour!

robotswantdata 10 hours ago||
WER is not good enough for professional use, fine for hobby projects
amelius 10 hours ago||
Sounds great if you could use it as a server. Otherwise, why would you want this?
etienne_l 10 hours ago|
[dead]
vladimirralev 10 hours ago||
Any info on the param size and quantization of each part and in total?
mikemzz1 11 hours ago||
interesting approach. But training ctc head on top of gemma's audio encoder will most likely produce better results and faster inference
Shershah1024 9 hours ago|
That's close to what this is: the reader is a CTC head on the audio encoder, but its output words are EmbeddingGemma's own text embeddings for ~229k words, instead of a separately learned vocabulary. I also tried a plain letter-level CTC head on the encoder. It did a bit better on clean audiobooks (7.0 vs 8.2 WER at the time) but much worse on harder audio like a lecture (16.5 vs 10.8), mostly from misspellings.
spiderfarmer 10 hours ago||
So Marco Arment can sell some more of his Mac Minis.
Aboutplants 9 hours ago|
First thing I thought of lol
guigoguedes 6 hours ago||
Does it work for portuguese?
jdw64 10 hours ago||
It would be nice if Korean also worked. I’d like to experiment with a Korean model too.
Imustaskforhelp 11 hours ago||
This has come at just the right time.

Does this work for Hindi or other non English languages though?

I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files

This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.

I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it out!

I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing

An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.

[0]: https://huggingface.co/parismitaglobalsolutions/indicconform...

1970-01-01 10 hours ago||
I've been using Whisper with great results; GPU acceleration gets the job done, nothing leaves my box, and the medium model is overkill.

https://openai.com/index/whisper/

https://github.com/openai/whisper

Imustaskforhelp 10 hours ago||
Yeah I will have to try whisper as well. I didn't try it because I wanted to tinker more and also I had thought that whisper was an older model but I think that if it gets the job, then it gets the job done and I will be trying it out on the text that I mentioned, thanks for mentioning it as I will give it another go :-D
tinykerneldev 10 hours ago||
TrueScribe supports tranlating Hindi transcripts in both directions. Not free though.
bbor 7 hours ago|
Ok let's talk about the really important stuff here: that model name is atrociously long for such a game-changing model, and "embedding" is already an awkward phrase. I propose... Embedda?
Shershah1024 5 hours ago|
He he. Thanks for the suggestion. Just wanted to show that an embedding model can transcribe. Maybe we should call it emscribe :D
More comments...