Are there models I'm not aware of that are trained for this task? Taking audio in an unknown language, and rather than identifying the language, just transcribing the sounds to IPA? That would not be useful to most people, but it would be a Godsend to many, many linguists working with minority languages around the world.
Thanks for the pointer; I'll check out that model and see if it handles the "slowly and carefully" type of recording better than the "natural speaking" type. (And depending on what kinds of errors the model makes, even the recordings where it makes errors can prove useful: for example, a linguist studying regional variations in speech would want the model to produce the IPA for "gonna" rather than "going to").
Example: Schwabisch is wild and would be phonetically transcribed very differently from Hochdeutsch which is its ostensible language progenitor (technically more a cousin than an ancestor in the lineage of language evolution), but if the goal is merely to focus the model purely on phonetic transcription then you can add additional post-processing layers which map sounds to core concepts shared across dialects for actual translation. But I like your idea of interacting with the intermediate elements to familiarize yourself at least with the phonetic patterns, we humans are still thinkers enough to infer patterns of grammar and semantics from these building blocks just as we have done for the entire history of the species/lineage before written representations of language came along (relatively late -- evidence of script cropped up only once civilization had centralized to a sufficient degree to make economics non-local and non-trivial).
tl;dr the big words: it's not til you collect enough spoken samples of the dead(ish/dying) language being spoken that the local idiosyncracies are discovered, luckily linguists are smart enough to probably anticipate and certainly post-process language snippets to grasp the common structures for this or that given language.
That's true if you mean "very rarely spoken" literally, as in even the native speakers don't get to use it very often. But many languages aren't widely spoken (such as only in a certain geographical area, which sometimes is only a single village, or other times a small number of villages). But inside that area, they are frequently spoken. And you might be surprised how many of those small-geographic-area languages still have distinct dialects.
For example: my wife (a linguist) did her master's thesis on the pronunciation of a language with about 7,000 speakers, and identified how many distinct dialects there were. (Which is why I know a little bit about this). She recorded native speakers from all 13 (I think it was 13, but it might have been 14) villages where the language was spoken, and found five different dialects, which she grouped into two "main" dialects. (Think American vs British in the English language, with subdivisions into Midwest, New York, and New England accents and so on, and you'll have the right general idea — though these dialects were closer to each other in sound than Midwest vs New York). I'd have to go reread her thesis to give you any more details. But this was a language that was only spoken in a small geographic area, but it was frequently spoken, because that was the main language of those villages. (The country's official national language is what the kids learned in school, but some of the people, mostly those 60 years old or older, hadn't gone to school, because the first government school in their area was only built 60 years ago -- so they only spoke their minority language, and not the country's language, and their kids had to translate for them if they had to leave their village and go shopping in a major town).
1. ZIPA https://github.com/lingjzhu/zipa
2. POWSM https://huggingface.co/espnet/powsm
3. PhoneticXEUS https://github.com/changelinglab/PhoneticXeus
I would be curious to know how to help people to use these models, or what kind of tasks they could be applied to.
Or maybe linguists are actually interested in having maximally faithful IPA representation and manually normalizing it? You are clearly way more knowledgeable about that topic than I, so I'm curious what you think.
Linguists studying the sounds of a language, its phonology, often want to find "minimal pairs", words that differ by only a single sound. For example, din and tin in English. You record a native speaker saying both words, and telling you their meaning, and then you play back either recording A or recording B to other native speakers and ask them which word it is. If they can identify the word every time, then you've found two sounds that are meaningfully distinct in this language. (Some languages don't distinguish the d and t sounds, but English does). But if the native speakers go 50/50 on which word it is, or ask to hear it in a sentence for clarification because it could be two or three different words, then you've found a pair of sounds that this language does not distinguish. (Note that you're playing the words in isolation, because sentence context might make it obvious which one it is, e.g. you can't tell if an English speaker is saying their or there until you hear more words of the sentence).
So yes, the linguists that I know (who, again, are not necessarily a representative sample) are interested in as faithful an IPA representation as they can get, because that inconsistent transcription will give them many clues about the language. It still all has to be checked, because that switching back and forth between aspirated and unaspirated p (for example) could have been an artifact of a poor-quality microphone not picking up the aspiration, or a windy day causing aspiration sounds that the speaker never said, or the speech-to-text model making a mistake. But I watched my wife listen to the same two-second recording on loop over and over, trying to be certain of which sound she was hearing in the middle of the word. Double-checking the output of the model would (in most cases) only require listening to the audio once or twice, not half-a-dozen times like she typically did while researching her thesis. At the time, LLMs were not really a thing yet, but if she were doing her thesis today I bet a speech-to-IPA model would have saved her quite a lot of time — but only if it output every distinction, even the ones not meaningful in the target language. The "maximally faithful" representation, as you put it.
Isn’t the recording itself “more ideal” than any representation of it?
The main benefit that I see (not being a linguist myself and seeing it second-hand, as it were) is saving time; it's quite a lot of effort for humans to listen to an audio recording and write it down in IPA. I didn't realize just how much effort was involved until I watched my wife researching her thesis, listening to the same recording over and over and over.
Also, there are possibilities for automated translation that the IPA transcription opens up. Speech-to-text models need different training than text-to-text translation models. The other day one of the linguists I know sent me a message, saying "I have a lot of recordings in language X" (I forget which language, but it doesn't really matter which language it is) "and I want to transcribe them. Is there a way to use AI to do that?" I thought it through, and realized that recording-to-IPA followed by IPA-to-correct-alphabet would be a way of dealing with the combinatorial explosion (according to https://www.ethnologue.com/insights/how-many-languages/ there are close to 7,200 languages spoken in the world, though "spoken" may be a misnomer since that number includes sign languages). There's no way anyone can train a model to convert about 7,000 languages from spoken form to written form; that's just way too many possibilities, and I suspect that audio models are necessarily large because audio is more complex information than written text. (If I'm wrong on that, someone please inform me). But although the human mouth (and throat) can produce dozens and dozens of different sounds, that's a finite number. (I think it approaches a couple hundred, but that's still a lot less than 7,000). So the number of distinct sounds and their IPA equivalents that the model would have to distinguish is smallish. Then another model could be trained on converting IPA to the alphabet (and spelling) of the world's 7,000-ish languages — and that would be a lot simpler than sound-to-spelling.
You're right that the recording is "more ideal" than any representation, and linguists will always archive the recording (in a lossless format, whenever possible, though if the original recording was done by a non-linguist who recorded it in MP3 then there's not much point in storing a WAV) alongside their transcription. This allows anyone else who might be doubtful on a point to go listen to the original recording and say "Yeah, Smith got the transcription wrong on that word. That's not an alveolar click, it's a velar click." But when you're writing a paper, you don't stick audio recordings into your document, you write down transcriptions. An audio recording without a transcription is not nearly as useful to linguists as having both. (EDIT to add: And such transcriptions, when you're writing papers about the sounds of a language, are pretty much always in IPA as well as in the language's standard alphabet. IPA isn't just a means to an end, it's also an end in itself. So even if a speech-to-text model existed for the language my linguist friend wanted to transcribe, he would still also want an IPA transcription as well, 90% of the time.)
IPA seems very comprehensive from my amateur perspective, but apparently a lot of modern linguists still extend it or roll their own.
Noob question: How do you think about funding from a foundation(i have no clue if you need it or not, I do hope you have a way to get paid one way or another because handy is amazing) for maintenance of this? if you did or were going to get paid by asking for maintaining such a project what might be the kind of organizations you would look for to get supported and how would you do it?
Certainly I am very lucky that quite a few people donate to Handy, and also some people and organizations who sponsor the work I do
To be honest I just love contributing to open source and wish to continue to do so. So anyone who supports this is good to me. Organizations which believe in OSS and push it forward are typically most aligned with me
Of course you can always email me (contact@handy.computer) and we can discuss in more detail
> Dictation sends information like your voice input, contacts, and location to Apple when necessary for processing your requests.
I want to open an office document, for example, and start talking. And I want the software to continuously type what I am saying at the cursor with minimal latency. The continuous part is crucial. Many software will paste whatever I said after I have stopped recording, but that is not useful.
I often prefer to dictate everything in my head about a particular thing for 5–10 minutes and then go through it afterward. I find that much more useful because it doesn’t break my thought process the way continuous transcription does.
I’m planning on having it as a first class feature of the app too just too many other issues to work on first
A while ago, I auditioned about 10 different STT apps on my Mac, with this realtime/streaming transcription as a goal. I failed to find that feature in an app I was happy with, but settled on Handy as the best option otherwise. So if Handy adds this, it will be perfect!
It may be easier than expected in some way since we already emit events for the live overlay, so it could be as small as a function call, but I don’t know the code path well enough from memory and what complexities it has. Probably with the Tauri context and a bit of other mess we have as this bit of code has gone through a lot of pain
In fact, it cleans up the entire paragraph that you just said, and even if you have meandering thoughts, it cleans those up too.
Actually, this above statement was fully dictated with iOS and it added all the punctuation automatically, so I think that iOS is also doing some of this natively. In fact, I’m on the iOS 27 beta and it seems to be doing an even better job of correcting itself and correcting earlier words and adding punctuation too.
But in this day and age it’s easy enough to at least write the iOS and Android versions. But maybe not dealing with the play store.
It may have been an early issue but with no obvious way to interact and report the issue and, eh, Google’s general attitude around customer satisfaction, I just gave up and deleted it again.
So I know it's possible. I just want to integrate this with the paste-at-cursor feature that these apps have. I imagine the app would have to create a virtual keyboard and use backspace or arrow keys to go back and change things.
I believe the built-in iOS dictation already does this.
Same with punctuation, you could determine that a comma belonged in a certain place once you have enough words.
So in general this definitely works. Handy is just missing the feature to insert these streamed words into the app where the cursor is.
It really depends on how one uses transcription.
For example, I really value being able to open different windows, and look at graphs, or scroll some data while I'm dictating, because it can help me with providing some support information for what I'm saying.
Some apps can even take into account things you copy or look at as part of the transcription's context to improve the results [0].
[0]: https://superwhisper.com/docs/common-issues/context#types-of...
However the accuracy of the real time models is poor, so I did a second pass with a higher accuracy model before committing the text.
It looks like the rust bindings have streaming examples so hopefully there is a nice solution here.
Nice. Here's the Python one: https://github.com/handy-computer/transcribe.cpp/tree/main/b... - looks like it's not yet available as a binary wheel on PyPI with the dependency included (the library on PyPI right now uses ctypes to call a separately installed library) but that's planned for a future release.
If there’s any issues or improvements on the bindings I would love help to make the DX the best it can be
It's a nice reminder: You can use AI to slop cannon at maximum speed, or you can use it to scale your ambitions and build something more rigorous and lasting than ever before.
I'd build Transcribe.cpp into the apps I maintain, but I feel like this functionality should (generally) be integrated into the OS or "everywhere" via an app like Handy.
I agree this should be everywhere and I hope to distribute libtranscribe some day properly so it is more a system library! It will take time to stabilize but I think we can get there
> I think as we look forward to the future, more inference will start happening locally for one reason or the other. This brings the distribution story front and center. In order to have more applications running inference locally, we need to make running inference easier.
This makes these projects so much more trustworthy and easier to approach:
> Were any of the words here written using AI? Nope. They came from my mouth or my fingers.
>> Were any of the words here written using AI? Nope. They came from my mouth or my fingers.
I have to push back on this a bit, as I believe (quite strongly) that we're shaped by the tools we use; text-to-speech LLMs are still LLMs, and generally their mistakes are shaped by the expectations inherent in their training. This, in turn, shapes the words that appear on the screen. For those who regularly use them, you then learn which word sequences are likely to be accurately transcribed, and this definitively becomes part of your thinking process. Over time, the LLM becomes tangled into your thinking; the use of AI, even in this way, very much can and often does shape the resulting words.
1. https://www.poetryloverspage.com/poets/tyutchev/silentium/li...
Looks like it's using IBM's Granite-Speech-4.1-2B-Plus https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plu... and/or MOSS-Transcribe-Diarize https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
I’m not sure how many specific models will be supported as the library is more focused on transcription specifically. But the models which support diarization natively must be supported I think. And parakeet multitalker was the primary driving force for this change
The github page doesn't seem to go into depth on these orthogonal topics. May have missed it.
For example: from audio samples tagged with names