| Building a speech-based reading tutor for kids, and the hardest part hasn’t been the ASR accuracy itself — it’s been distinguishing “the model misheard this” from “the child actually mispronounced this,” which are very different signals for a tutoring app (one needs a retry, the other needs feedback). |
|---|
| We’re using Riva for real-time recognition, fine-tuned on a children’s speech dataset, but confidence scores alone don’t cleanly separate the two cases — a genuine mispronunciation of a phoneme often produces a similar confidence pattern to background noise interference. Right now we’re layering a phoneme-level comparison against the target word on top of Riva’s output, which helps but adds latency we’re trying to claw back elsewhere in the pipeline. |
| Has anyone built something similar — ASR for language-learning or pronunciation feedback specifically — and found a cleaner way to separate “recognition uncertainty” from “genuine pronunciation deviation”? Also curious whether others have found meaningful gains from further fine-tuning versus this being more of a post-processing problem no matter what. |
| This is part of an education app we’re building at Dev Technosys — the phoneme-comparison approach has gotten us most of the way there, but curious if there’s a more elegant solution we’re missing. |
Hi @Snehadevtechnosys — this is a really hard problem and confidence scores are indeed the wrong signal to rely on. A few things that tend to work better for separating “model didn’t hear” from “child said it wrong.”
**1. Use word-level time-aligned confidence, not utterance confidence**
Riva can return word-level timestamps and confidence. A genuine mispronunciation usually shows up as:
- Low confidence on the specific phoneme/word that was mispronounced
- The rest of the utterance is clean
Background noise or poor audio, on the other hand, tends to produce scattered low confidence across multiple words or whole phrases.
**2. Add an acoustic quality gate before the phoneme comparison**
Check SNR, clipping, or silence ratio first. If the audio quality is bad, flag it as “please repeat” before doing any pronunciation analysis. This removes a lot of false positives from the phoneme layer.
**3. Combine Riva output with a pronunciation model**
Instead of comparing the recognized text to the target text, compare the *phoneme posterior probabilities* from Riva (if available in your pipeline) against expected phonemes. A mispronunciation will show a substitution/deletion at a specific phoneme position; a recognition error will show a mismatch without a clear phoneme-level pattern.
**4. Use children’s speech augmentation, not just fine-tuning**
If you already fine-tuned on children’s speech, the next gain usually comes from data augmentation — pitch shifting (kids vary widely), simulated room reverberation, and additive noise. More data often beats more model capacity here.
**5. Two-stage UX**
For a tutoring app, I’d consider a soft approach:
- First attempt: give the child the benefit of the doubt
- If the same word fails twice with similar phoneme deviation → pronunciation feedback
- If it fails with wildly different recognized text → “I didn’t catch that, please repeat”
The latency you’re adding with phoneme comparison is probably unavoidable for good feedback, but you can make it async to the UI so the app still feels responsive.