Bilingual voice agents: why Spanish-first is harder than you think

I served seven years in the U.S. Air Force as a Defense Intelligence Agency Spanish Linguist before going to law school. I've practiced bilingual immigration law for over a decade. And I still underestimated how hard it would be to ship a voice AI agent that handles Spanish callers as well as it handles English callers. This post is what I wish someone had told me before we started — the things that look obvious in retrospect, the failure modes that only surface in production, and the architectural decisions that turned out to actually matter.

When you evaluate a bilingual AI voice product, you usually get one of two things. Some vendors offer a 'Spanish option' that's actually a separate routing branch — the caller hears 'press 2 for Spanish,' a different agent picks up, and there's no shared context. Others run the same agent prompt translated through machine translation at runtime. The second option is worse: you get an agent that sounds like Google Translate read aloud, with idioms and tone that signal 'this isn't really for you' to native speakers.

Neither approach handles the actual reality of bilingual caller bases, which is that callers code-switch. They start in English, drop into Spanish for a phrase that's easier to say in Spanish, switch back. They give a name in Spanish and an address in English. They expect the agent to follow without making a fuss. Most products don't follow.

We made language a property of the agent's state, not a routing decision. The same agent answers every call. It listens to the caller's first turn, detects the language with a model, and continues in that language — Spanish-only callers get the entire call in Spanish, English-only callers get the entire call in English, code-switchers get the agent following their lead. When the caller switches mid-call, the agent doesn't restart or apologize; it just continues in the new language.

Key insights from this article

The Spanish prompts are not translations. They were written in Spanish from scratch, with the qualification questions phrased the way I'd phrase them in person to a Spanish-speaking client in our Charlotte or Orlando office. The intake decision tree is the same; the wording, register, and idiomatic flow are native.

• ✓Same agent, same context, both languages — no routing branches
• ✓Spanish prompts written in Spanish, not translated from English
• ✓Language detection runs every turn, so code-switching is handled gracefully
• ✓TTS uses native Spanish neural voices tuned per regional accent
• ✓Recognition models cover Caribbean, Mexican, Central American, and Andean Spanish

Spanish recognition is harder than English recognition for one specific reason: regional accent variance is much wider in Spanish than in US English. A native Cuban speaker, a native Mexican speaker, and a native Argentine speaker have noticeably different phonetic patterns, and a model trained primarily on Mexican Spanish will misrecognize Caribbean callers in subtle but conversion-killing ways. We spent 3 weeks tuning recognition for our specific market mix (Charlotte, Orlando, PR) — heavily Mexican, Cuban, Puerto Rican, Central American. The general-purpose models were almost good enough; the gap was on names, addresses, and immigration form numbers.

Names and proper nouns are where most multilingual STT systems quietly fail. 'Joaquín García' is not a hard name to a Spanish speaker; it's a hard sequence of tokens for a recognizer trained primarily on English. Our caller base is full of Spanish proper nouns. The recognizer would render 'Joaquín' as 'Walk-in' or 'Hawking' on first calls. We added a custom vocabulary biasing layer with the most common 5,000 Hispanic given names and surnames, plus the practice-area-relevant terms (USCIS form numbers, immigration program names, Spanish-language legal terms). Recognition accuracy on names jumped from roughly 70% to over 95% on real-call samples.

Bilingual law firm intake has cultural patterns that an English-speaking AI engineer wouldn't necessarily think about. A few examples that surfaced during tuning: extended-family decision-making is common — the caller is often not the person who will eventually retain. The agent has to capture both 'who I'm calling about' and 'who I'm calling on behalf of' as distinct entities. Mixed-status households — a citizen child, a permanent-resident spouse, an undocumented sibling — are normal in our caller base, and the intake flow has to handle the matter cleanly without forcing the caller to disclose family-member status if it's not needed for the specific matter.

The reverse is also true. Sometimes the caller assumes the agent knows things that an English-speaking intake system would have to ask. 'Mi residencia se vence' (my residency is expiring) is a complete-enough description for an immigration attorney to know what the conversation is about — the agent shouldn't waste five turns asking clarifying questions when the caller has already told it the matter type.

We tested four Spanish neural voices in production. The differences in caller behavior were not subtle. Voices that sounded native and warm got longer call durations and higher consult-conversion rates than voices that sounded technically Spanish but read with English-pattern intonation. We're now on a single Spanish voice tuned for our market that we've A/B'd for 90 days. The English voice is a separate selection, picked for similar warmth in English. The two voices don't sound the same — and that's fine. Callers don't expect them to.

Two things. First: invest in custom vocabulary biasing on day one, not after the first month of production calls. The names and form numbers were predictable; we should have biased for them before launch. Second: write the Spanish prompts before writing the English prompts. We did it the other way and ended up rewriting Spanish prompts to match the English logic — which led to translations creeping in. Starting in the bilingual primary-language and translating to English is a better discipline.

Ready to Transform Your Law Firm?