Posted by aarushyadav 2 hours ago
Here's a video of Dave from YC trying it out: https://www.youtube.com/watch?v=9QdX1eObVYI
(Edit: you don't actually have to move your lips or even open your mouth for this to work! Since the RF can see inside your mouth the system can be fine tuned on "internal speech". This isn't the automatic tongue movements that occur when "thinking" words to yourself, but more like mouthing with your lips closed. It's a bit weird to do, but with practice, we have been able to get it down pretty well.)
Arnav and I were studying CS at Georgia Tech at the beginning of the year, with no experience in silent speech or sensing. We were both AI-native builders and power users of Wispr Flow. What bugged us was not being able to use these tools in the library (or an office) or in public. At Stanford’s TreeHacks in February, we decided to build the most technically ridiculous thing we could think of. We’d read about AlterEgo, a silent speech system out of MIT’s Media Lab, and wanted to recreate it in 36 hours. We built a crude prototype, sticking EMG electrodes onto my face:
https://drive.google.com/file/d/1mwhv5HZMz3UTevdrgwS5O7g2jWV...
We got it to kind of work at a letter level in a very janky way. It wasn’t good enough to win the hackathon, but convinced us to go all in. The first time I typed a sentence without anyone noticing, I felt the magic of the technology.
We quickly realized EMG (reading muscle electrical signals) wouldn’t work:
- Signal changes due to sweat
- Electrical interference
- Placement variability
- Most importantly, no one wants to stick electrodes onto their face!!
So we switched to RF and built the glasses you see now. They emit radio waves, similar to Bluetooth, that pass through your face. As they cross the tongue, lips and soft palate, they’re modulated by silent mouthing. We train foundation models on this data to decode these perturbations to text.
Despite prior academic research, getting this to work was a huge engineering challenge. First we built an extremely low noise RF sensing stack. Then we had to place antennas so their fields overlapped maximally with the articulators. Our first attempt on glasses failed completely, so we reverted to old habits and taped antennas on our face:
https://drive.google.com/file/d/1ks0dS0eZiTV9EqSFsyB_w5BBjpc...
Since taping antennas consistently in the same spot and angle was hard, we created this weird but effective head rig to hold them in place:
https://drive.google.com/file/d/1rhzNFpdDzr7oHk0isFY70dkoV4C...
This worked, but obviously wasn’t a product yet. So we worked deep in simulations to design glasses that could hold the antennas.
That evolved into our design now, with antennas embedded within the frames. Once we established there was a signal, the next hurdle was cross-user generalization. Everyone’s facial anatomy, and hence RF signal patterns, are vastly different. We tried algorithmic workarounds but realized the only lever was to massively scale pretraining data.
So that’s what we did. We set up booths in our office and paid people to read prompts wearing our glasses, to pretrain our models on. This produced a beautiful scaling curve, with errors falling quickly as we scaled our dataset and cross-user performance becoming genuinely compelling (as you see in the video above; Dave only had a short 20-sentence calibration).
We always heard about building things from first principles but never understood it before this. We tried tackling it top down but completely failed. Only when we broke each component (the antenna, electronics, ML) down to its fundamentals to understand and optimize it did we get it to work really well.
Over the next few months we’re continuing R&D to turn our prototype glasses into the slick product on our website and ship in Q1 2027. We’re also scaling up our dataset to improve performance on edge cases.
For anyone who loves voice dictation but hates using it around people, we’ve opened preorders for our first batch of 500 units, with a $200 discount off the final price. Allocation is running out quickly, so get yours at https://subvocal.co/preorder!
Second, how tough is this signal processing? Will it be done on-glasses, or through an app? Third, what's the equivalent of a wake word for these? I'm reminded of the Apple Vision Pro radar to see if your fingers are clicking. I could imagine a lot of modalities where I want to just start talking and have things happen, and a lot where I don't want the glasses to pick it up. Fourth, I think a cool voice change demo would be rad -- not sure what latency is like, but if you could get it end to end under like 250ms you could probably have someone exaggerate the mouth movements and pipe out audio of them talking like someone else. Which would be another really cool demo.
Super cool. You're going to be working 80 hour weeks for the next year, I'm excited to see what you put out!
The decoding is done via a companion app on your laptop or phone in the cloud, but end to end latency is still quite fast, under 500ms usually.
To begin with we don't plan to have a wake word, but rather a trigger key on your connected device that you're entering text into (pressing Alt for example), similar to Wispr Flow. We're doing some R&D into the silent speech equivalent of voice activity detection, so we have a low power basic RF detection always-on, not enough to decode speech from but enough to detect rough movement, and when it detects speech-like activity beginning to start we can trigger full sensing within tens of ms. This would likely ship as a software update a few months after shipping, though.
The voice change demo is something I've thought about actually, but we never got around to doing. Might film it soon :)
Have fun building this! It's worth doing.
That's exactly where my mind went. Something I've wanted ever since I read that book a couple decades ago.
This is just really groundbreaking and I know the research has been out there for a little while but to actually implement is a whole nother ball game.
I think the use cases for this will expand as the product ships and gets out there!
"In the long, windowless hall, with its double row of cubicles and its endless rustle of papers and hum of voices murmuring into speakwrites, there were quite a dozen people whom Winston did not even know by name"
Maybe I could have figured that out from the video but for folks who can’t watch the video it’s quite hard to discover what you’re building.
1. Insanely cool
2. Gives me serious "whispering earring" vibes. Imagine having a full on conversation with an AI via subvocalization and listening with ear bud to guide you throughout the day. https://gwern.net/doc/fiction/science-fiction/2012-10-03-yva...
There is a reason secure activity happens within secured rooms built to a specific standard https://en.wikipedia.org/wiki/Sensitive_compartmented_inform...
I don't think it is an intuitive action at all and I became self-conscious about not making a sound rather than talking like I normally would.
I suppose it would require some "getting used to."
Regardless, I think it is a cool idea and a real problem. Sometimes I don't want to type nor talk out loud so I find myself whispering at my phone like I'm telling it a dirty secret.
--
As I'm writing this, I think this kind of product might not hit the market for consumers, but I'm seeing this used by Flock like companies to spy on people's speech, so kudos on making machine lip reading a thing!
I've found the opposite. Typing feels much more "deliberate", and of course you can hit backspace but you can't un-speak a word. With prompting, I've found that, in general, speaking leads to lower-quality results than typing.
> with many people
Not everyone. I, at least, definitely overthink what I type, and while I end up with nice-ish sentences in my prompts it takes a lot longer than a train of thought that the AI parses correctly most of the time anyway.
I suspect this has to do with the nature of the work you're doing; I dunno, just count me firmly in the camp of people who only rarely feel like I can express a thought sufficiently for coding agents with their voice.
But I guess thinking is going out of style anyway.
That said, I know a lot of people prefer talking to their computer, especially when their computer is a phone. So there is definitely a place for stuff like this.
But happy to see people get more options.
This reminds me of Eternity's End by Jeffrey Carver and the implants they had. Released in 2000 but the more time goes on the more the neural implant technology is coming about. To be fair I think Enders Game had similar tech to detect subvocal speech like this product.
Same. I'm often told I talk too fast, but I feel I communicate slowly because a lot of my fast speech involves halting and backtracking and such.
I can easily type 100WPM+, but I trivially blast out a few paragraphs of perfectly adequate text in a few seconds and then spend a few minutes re-editing it.
So this could take "noise canceling" to a whole new level if you could somehow "speak" across a loud bar or across the world, for that matter. Being integrated into something like Signal or similar end to end encrypted platform would be interesting as well.