The voice that lives between two languages
You upload the audio and wait. Your speaker is a second-generation Mexican-American from San Antonio, code-switching mid-sentence, her vowels shaped by Spanish, her rhythm shaped by Texan English. The transcript comes back with "espresso" where she said "express," gaps wherever she slipped into Spanish, and a confident misfire every time her intonation follows Spanish stress patterns inside an English sentence.
Not a glitch. A structural problem baked into how these systems are built.
What the model is actually doing when it hears you
Most commercial speech-to-text systems, including Whisper, Google's Speech-to-Text, and Amazon Transcribe, are built on sequence-to-sequence neural networks. The model listens to short audio frames, roughly 25 milliseconds each, and builds a probability distribution: given this sound, in this context, what word comes next?
Context is the operative word. The model doesn't just hear phonemes in isolation. It leans on a language model baked into its training, constantly predicting what word is statistically likely to follow the last one. That's why it handles fast, mumbled speech with surprising accuracy when the speaker fits the training data. The language model fills gaps.
But that same language model becomes a trap when your speaker sits between two languages.
The model has to pick a lane. It was trained predominantly on monolingual audio, and its probability engine is calibrated for speakers who sound like the majority of its training corpus. Accents that blend phonology from two distinct language systems don't fit cleanly into either distribution. The model hedges, and hedging almost always means defaulting to whichever language it has more data for. This is not a close call; it's the only move the model knows.
The spectrum problem, specifically
An accent isn't a fixed point. Think of it less like a flag and more like a hand on a dial, one that shifts depending on who she's talking to, how tired she is, or whether she's describing something she originally experienced in Spanish. Her English vowels are slightly backed, her consonants sometimes voiced where standard American English would devoice them, and her prosody follows Spanish-influenced patterns even when every word she says is technically English.
For a human listener, this is unremarkable. We adjust without thinking about it.
AI transcription doesn't adjust. It applies a fixed acoustic model trained on a fixed corpus. When a sound doesn't match the expected phoneme for the predicted word, the model either mishears the phoneme or reinterprets the whole word. A backed vowel in "bad" might get transcribed as "bod." Spanish-influenced intonation that rises on a statement rather than falling can confuse the model's sentence boundary detection, chopping one long sentence into two fragments.
The result isn't random noise. It's systematic error, the same speaker producing the same kinds of mistakes in the same phonetic environments, every time.
Where training data actually fails these speakers
This part is worth slowing down for, because the failure isn't mysterious. It's arithmetic.
Whisper was trained on roughly 680,000 hours of audio. That sounds enormous. OpenAI noted that a significant portion was English audio scraped from the internet, and internet audio skews heavily toward speakers from the United States, the United Kingdom, and Australia, specifically toward people who record podcasts, YouTube videos, and audiobooks. That demographic is not a representative sample of how English is spoken by the 1.5 billion people who use it.
Now consider Chicano English, or Singlish, or South African Indian English, or the Hindi-inflected English spoken across urban India. Each is a fully formed, rule-governed variety with its own phonological patterns. None of them are edge cases of "bad" pronunciation. But they are statistically underrepresented in the training data, which means the model's confidence scores for those phoneme-to-word mappings are lower.
Lower confidence means more errors. More errors means the model falls back on its language model more aggressively, predicting what it expects rather than transcribing what it hears. This feedback loop makes systematic bias self-reinforcing, and that is a design failure, not an unfortunate side effect.
The gap shows up clearly in practice: Priya, a software engineer from Hyderabad whose English carries Indian prosodic patterns, and Mark, a software engineer from Ohio, both sit for the same 45-minute interview with the same recording setup. Mark's transcript comes back at roughly 95% word-error-rate accuracy. Priya's comes back at 78%. That 17-point gap isn't about clarity or volume. It's about whose voice the model was optimized for. Do you think Priya's clients notice when her auto-generated captions are riddled with errors and Mark's aren't?
What actually helps (and what doesn't)
Language selection is the first lever, and most people ignore it. If your speaker is code-switching between Spanish and English, setting the transcription language to "English" and hoping for the best is the wrong call. Some tools, Whisper included, support multilingual transcription that can detect language switches mid-audio. Imperfect, but substantially better than forcing a monolingual model onto a bilingual speaker.
Custom vocabulary lists help with proper nouns and domain-specific terms. They don't fix phonological mismatches, but they reduce one category of error.
Speaker adaptation, fine-tuning a model on a specific speaker's audio, is the most powerful fix and the least practical one for most workflows. You need dozens of hours of clean, labeled audio from that speaker. A research-grade solution, not a Tuesday afternoon one.
Slowing down doesn't help as much as people think. The model's errors aren't caused by speed. They're caused by phoneme mismatch, and a speaker with a Spanish-influenced vowel system will produce the same vowels at 90 words per minute as at 120.
The honest answer: post-editing is still necessary. For accents on a spectrum between two languages, no current commercial tool achieves the accuracy it achieves for mainstream American or British English. Build that expectation into your workflow rather than assuming the transcript is clean.
The fix that's actually coming
The most promising direction isn't bigger models trained on more of the same data. It's targeted data collection from underrepresented speaker communities, combined with self-supervised learning techniques that can adapt acoustic models with far fewer labeled examples than traditional fine-tuning requires.
Some research groups are already doing this. Mozilla's Common Voice project has been collecting voice data in over 100 languages, including regional varieties, specifically to counteract the demographic skew in existing corpora. Models trained with that kind of intentional diversity show meaningfully lower error rates on non-mainstream accents, sometimes cutting the gap between high-accuracy and low-accuracy speakers in half.
Half is progress. It's not parity.
The deeper issue is that accent diversity isn't a bug to be patched. It's a feature of human language that reflects migration, identity, and history. A tool that works well only for speakers who sound like a particular slice of internet audio isn't a neutral tool; it has a thumb on the scale. The transcription model doesn't know that. It's just doing math on phonemes. But the people building those models, and the people choosing which ones to deploy, do know it, and that's where the actual decision gets made.