You can just embed those new words and they get added to vocabulary. It is fairly simple. One risk is - When a new word is very close to a known one, it tends to fall to it.

Yes. Planning to train a multilingual reader. Have not done a prototype yet.Will do one once I have the right dataset

It's not being played through a desktop audio stack, they're operating on the waveform decoded from the audio file. There's no constraints relative to the "natural speed" of the source, just like for video transcoding, it's limited by compute.

during transcription, nothing is actually "playing" the file in the way a human would to listen to it. the audio is stored as a sequence of bytes representing the amplitude of the physical sound waves over time (the recording device measures, or "samples", the amount that its microphone is vibrating and records that amount, some tens of thousands of times per second) with some metadata and compression applied so that other software knows how to interpret the bytes correctly. this can be given to an audio program to convert the bytes back to physical sound waves using your speakers, but it's just bytes at the end of the day, so any software that can read bytes out of the file can do so at whatever speed it wants, regardless of how long the actual recording is.

so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.

in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.

Probably significant parallelism. Split the audio into a lot of slightly overlapping portions to ensure the transitions between words are caught. I’m sure there’s some relatively simple ways to ensure you don’t double up on words.

No more than an AI media generator creating the media in realtime.

Is it low quality in the "It didn't match the meaning that was conveyed" sense, or in the "It didn't capture the exact words used" sense?

My reaction to seeing the project was that being an embedding model it'd be pretty good at conveying the meaning but not great as using the same terms, is that what you're seeing?

The benefit is that you can take input in one "mode" and extract output in another; that is, the input is audio mode and the output is text mode. The intermediate representation is the "embedding".

Multimodality seems to just help with everything

Correctly transcribing requires context too, no?

Only English for now. I dont see any roadblocks for at least the major European Languages(Not sure about German though with their compound words)