Quick Answer: Silent speech recognition with AirPods is possible only if head-motion data carries real signal about mouthed words. Subvox, an open-source macOS prototype, tests that using Apple's CMHeadphoneMotionManager, a dynamic time warping template matcher, and a 500-run permutation test. Whether it works depends on your own data, not on the demo.
Most people picture silent speech recognition as a camera pointed at your lips or electrodes taped to your face. Subvox asks a cheaper question: do the small jaw and head movements that come with mouthing words show up in the motion sensors already inside your AirPods? The answer it gives is more careful than yes or no, and that's the useful part. Here's what the project measures, how it decides, and where it breaks.
What the AirPods Sensors Actually See
The prototype reads data through CMHeadphoneMotionManager, Apple's CoreMotion interface for headphone motion. It gives you user acceleration, rotation rate, and attitude, but at a low rate. The README puts it at roughly 25 Hz on AirPods, which is far slower than a phone accelerometer.
That rate changes what you can expect. The README is explicit that the speech-detecting accelerometer and the inward-facing microphone aren't exposed. So you're not capturing vocal vibration. You're capturing mechanical side effects: jaw movement, head tilt, and small shifts in posture while you form words.
This is the first place most assumptions go wrong. People hear "AirPods" and imagine the device listening to bone conduction. What the software actually gets is a coarse motion trace, and that's a much weaker signal to build a recognizer on.
How a Mouthed Word Becomes Text
The decode pipeline has four stages, and each one is a place where things can fail:
- Segmentation. The app starts a segment either when you hold Space or when an energy trigger crosses a calibrated noise floor, with pre-roll and hangover so the start and end of a word aren't clipped.
- Feature extraction. Each frame yields nine values: three for user acceleration, three for rotation rate, and three for attitude change since the segment started. Each channel is mean-centered, and quiet lead-in and lead-out frames are trimmed.
- Matching. Each channel is z-scaled against the training set. A 1-nearest-neighbour classifier then compares the segment to each stored template using dynamic time warping with a Sakoe–Chiba band. The band limits how far the warping path can drift, which stops the algorithm from matching a fast utterance to a slow one in absurd ways.
- Rejection. A segment is rejected if its best distance exceeds max distance, or if the margin to the runner-up falls below min confidence.
The rejection step matters more than it looks. A system that always returns its best guess will confidently produce wrong words. Here's where the design earns its keep: a recognizer that says "I don't know" is more honest than one that guesses on every take.
Why Template Matching Instead of a Language Model
Subvox doesn't use a language model at all. That's a deliberate choice, and it's the opposite of how most modern speech systems work. Large models predict likely words from context, which helps when the audio is noisy but does nothing to prove the motion signal is informative.
Template matching is a blunt tool, and that bluntness is the point. With a small vocabulary and a few dozen takes per word, you can see exactly what the classifier is comparing. There's no hidden prior doing the work for it.
The trade-off is scale. A 1-NN template matcher needs you to record every word you want to recognize, in your own voice, in your own posture, on your own AirPods. It won't generalise to a stranger, and the README says so: templates are user- and session-specific, and re-seating the earbuds can shift the signal enough to matter.
The Test Most Demos Skip
Here's the counter-intuitive part. The popular way to show a motion-based recognizer works is to report its accuracy. Subvox treats that number as nearly meaningless on its own.
Consider a five-word vocabulary. Random guessing gets 20% right on average, so a leave-one-out accuracy of 30% sounds like progress. But with ten takes per word, fifty trials is a small sample, and some lucky runs will hit 30% even when the motion stream carries no information at all.
Subvox runs the same leave-one-out procedure on shuffled labels 500 times. That builds a null distribution, a picture of what accuracy looks like when the words are meaningless. If your real accuracy sits above the top of that distribution, with p < 0.05, the motion data is distinguishing your words better than chance. If it doesn't, the 30% was noise.
This is a permutation test, and it's cheap to run. Skipping it is how people end up believing a prototype works when it's only fitting random structure in a small dataset.
Which Sensor Channels Carry the Signal?
The Train screen lets you turn feature channels on and off and re-run evaluation. That turns a vague question, "does motion help?", into a testable one. The table below summarises what each channel captures and what can go wrong with it.
| Channel | Values per frame | What it tends to capture | Main risk |
|---|---|---|---|
| User acceleration | 3 (x, y, z) | Jaw and head displacement during word onset | Sensitive to walking, typing, or any body motion |
| Rotation rate | 3 (x, y, z) | Angular head movement across the utterance | Drift and noise at low sample rates |
| Attitude change | 3 (roll, pitch, yaw) | Net posture shift since segment start | Affected by re-seating and slow posture changes |
The practical move is to run Evaluate with each channel alone, then in combination. If accuracy holds with rotation alone but collapses with acceleration added, your acceleration channel is probably picking up incidental body motion, not speech. Here's where most people skip the ablation and report the best combined number instead.
Running Your Own Experiment
The project includes an experiment protocol that makes this kind of test repeatable. Start a Record Session from the toolbar or the Sensors screen. Each session writes to ~/Documents/Subvox/Sessions/<timestamp>-<condition>/ with four files:
motion.csv: every motion sample, withtin seconds from session start.audio.caf: microphone audio, if the mic was running.events.csv: every Space-held segment, with its label or prediction.meta.json: the condition, the prompt, and the audio start offset for aligning audio with motion.
The suggested design is a five-word script recorded under three conditions: normal speech, whisper, and silent mouthing, with ten repetitions each. Comparing the three tells you something useful. If silent mouthing performs close to normal speech, the motion stream is carrying the articulation. If it only works for whispered speech, you're probably picking up breath and vocal effort rather than mouth shape.
Here's the catch with this design. Repetitions from the same session are correlated, so a high score can reflect how you happened to sit that morning. Record across more than one session before you trust the result.
Known Limits and Failure Modes
Several limits in the README deserve more weight than they get in a typical demo:
- Re-seating shifts the signal. Templates recorded with one fit may not match a second fit. Re-train after you move the earbuds.
- Mic input changes the audio profile. Choosing AirPods as the microphone input switches them to the Bluetooth headset (HFP) profile. That's a different audio route, so calibrate with the mic configuration you'll actually use.
- Low sample rate limits resolution. At roughly 25 Hz, fast articulations get only a few samples each. Some distinctions that look obvious in your head may not survive the sampling.
- Templates don't transfer. A dataset recorded by one person won't classify another person's words. The
dataset.jsonfile lives in~/Library/Application Support/Subvox/, so keep it per user.
These limits don't make the project useless. They define what it can honestly claim: a measurement of whether this signal exists for you, in this setup, on this day.
Frequently Asked Questions
What is silent speech recognition?
Silent speech recognition is the task of turning mouthed words, with no vocal sound, into text. Systems usually read movement from the lips, jaw, tongue, or throat. Subvox tests a narrower version: whether the head and jaw motion captured by AirPods carries enough information to separate a small set of words.
How do I test whether AirPods can detect silent speech?
Record a small fixed vocabulary, such as five words, under normal speech, whisper, and silent mouthing, with about ten repetitions each. Train templates, then run leave-one-out evaluation and a permutation test with shuffled labels. Only trust the result if your accuracy clearly beats the shuffled-label distribution, with p below 0.05.
Why does accuracy need a permutation test?
Small datasets produce high accuracy by chance, especially with few classes and few trials. A permutation test shows what accuracy looks like when labels are meaningless, so you can judge whether your real result stands out. Without that baseline, a number like 30% on a five-word task tells you very little.
Can silent speech recognition work with any headphones?
Only headphones that expose head-tracking motion through an API can feed a system like this. Subvox requires macOS 14 or later and AirPods Pro or another model with head tracking, connected as the Mac's audio output. Even then, the templates are personal, so results won't carry over between users.
Silent speech recognition with AirPods is a testable question, not a settled one. Record the five-word script under silent mouthing and whisper this week, then check whether your accuracy clears the permutation baseline before you believe the result. If you're trying to figure out whether your own AirPods carry usable signal, pass this to someone who's been wrestling with the same question.