A customer calls to ask about an instalment payment. They begin in Mandarin, use the English product name, pause to check a detail and then correct themselves: “不是 credit card，是 instalment plan.”
For the customer, this is one ordinary request. For a voice AI system, it is several problems at once. It needs to recognize two languages over a phone line, preserve the correction and respond without making the caller start again. If the customer asks for an account-specific answer, it also needs to find the right information and follow the business rules for sharing it.
That is the difference between supporting multiple languages and handling a multilingual conversation. The first means a system can process more than one language. The second requires it to follow what a person means as their words, pronunciation and intention change in real time.
For the teams building or selecting a voice agent, the question is not just whether a model can transcribe a sentence. It is whether the whole system can hear the customer, keep up with the conversation and move the request towards a useful outcome.
Why code-switching is harder on a phone call
A customer does not announce a language switch before it happens. A caller in Hong Kong might use Cantonese for the conversation and English for a product name. In Singapore, a conversation may combine English, Mandarin and local expressions. Arabic-speaking customers may use a local dialect alongside English terms. These are examples of how people may speak, not rules about how everyone in a market communicates.
The difficulty is not only the number of languages. A borrowed word may be pronounced with a local accent; a product name may sound like an everyday phrase; background noise can hide the syllable that distinguishes one from the other. When the speaker corrects a detail, the system must recognize both the correction and what it replaces.
Telephone audio adds another constraint. Some traditional telephony uses narrowband audio sampled at 8,000 times per second. Compared with wideband audio, less speech-frequency detail is available to the recognition system. Noise, a weak connection or a speaker farther from the microphone can make a difficult word even harder to recover.
This is why performance on clean recordings alone tells a team little about how an agent will handle a busy contact-center call. The model has to work with the audio it receives, while the rest of the system has to recognize when the available signal is not clear enough for a confident answer.
From speech to meaning: getting the important details right
Automatic speech recognition (ASR) converts speech into a form the system can process. In a code-switched sentence, recognition may need to distinguish language patterns, local pronunciation and business vocabulary within the same utterance. An English product name spoken with a local accent should remain identifiable even when the surrounding sentence is in another language.
Dyna.Ai develops voice models for real conversational conditions, including mixed-language speech, accents and background noise. Its Voice Agent is designed to handle local speech patterns in customer conversations. That matters when a caller does not use the exact phrasing in a script, or when a product name is embedded in another language.
Recognition quality is often measured with word error rate (WER) or character error rate (CER). Both compare a transcript with a reference, counting substitutions, deletions and insertions. In multilingual speech, teams also need consistent rules for spelling, script and word boundaries before comparing scores. Research on Arabic–English code-switched speech illustrates why transcript representation can be challenging across languages.
An overall error rate is useful, but it does not show the full business impact. Mishearing an account type, an amount or the word “not” can change the request. An agent may produce a fluent reply and still answer the wrong question. That is why recognition and downstream understanding need to be considered together.
Understanding a conversation that changes direction
A transcript is not a complete service record. The conversation also needs to retain the customer's intent, the information already established and any changes the customer makes.
Consider the caller who corrects “credit card” to “instalment plan.” A conversation system needs to treat the second phrase as a correction, not a second unrelated topic. If the caller later asks to make a payment, the request to pay is a new action, separate from the earlier request for an explanation. Relevant context needs to follow the conversation without turning an earlier assumption into an instruction.
One common design is to maintain structured conversation state: values such as the selected product, requested action and confirmation status can be updated as the dialogue progresses. Dialogflow CX session parameters illustrate this general pattern. The implementation varies by platform; the principle is that a correction must be reflected in the information used for the next response or action.
Timing creates a trade-off. Processing partial speech can help a system prepare earlier, but an unfinished phrase may omit the very words that change its meaning. Retrieving general guidance while the customer is still speaking can be useful. Taking a consequential action before the customer has finished or provided the required confirmation is a different matter. Good conversation design separates early preparation from an authorized action.
Words also share the channel with pauses, emphasis and tone. A pause can mean the customer is thinking, not that they have finished. A change in tone may be a signal that the conversation needs attention, but it is not a reliable diagnosis of what someone feels or who they are. The system needs to respond to the conversation in context, including a clear request to speak to a person.
Natural speech depends on timing as well as voice quality
Customers do not take turns as neatly as a text chat interface. They interrupt, add a detail midway through an answer or change their minds. If the agent continues speaking over them, even an accurate response can feel like a failed conversation.
Interruption handling combines several decisions: detect that the caller is speaking, stop or adjust the outgoing audio, and interpret the new input in context. A design that reacts too slowly makes customers wait to correct it; one that reacts to every background sound or short acknowledgment can stop unnecessarily. The right balance depends on the environment and the conversations the service handles.
Streaming can reduce the wait between the end of a customer's turn and the start of a response. Instead of waiting for every stage to finish before starting the next, a streaming pipeline can process incoming audio and prepare output in overlapping stages. Dyna.Ai’s self-developed inference engine uses parallel streaming across speech recognition, language understanding and speech synthesis. The purpose is to keep the conversation moving while the system continues to process the incoming speech.
Speed alone is not the experience. The system must also manage what happens to audio already queued for playback when a caller interrupts. In a bidirectional Twilio Media Stream, for example, an application can send a clear message to empty buffered audio. Different telephony systems expose different controls. Stopping audio also does not undo an action already submitted to another system; that requires the appropriate business process.
The voice itself matters once the agent has something useful to say. Dyna TTS provides language-specific speech synthesis for Japanese intonation and business terminology, and Cantonese delivery for Hong Kong. Its Arabic voice supports Arabic–English mixed text and pronunciation guidance for local names and industry terms.
Naturalness is more than choosing a pleasant voice. Product names, loanwords, dates and amounts must be understandable in the language and context of the conversation. A response that sounds smooth but makes an amount difficult to understand can still leave the customer uncertain about what happens next.
From a good conversation to a completed task
Voice AI creates business value when the conversation leads somewhere. A service caller may need an answer, a sales prospect may want a follow-up, and a customer contacted about a payment may need an appropriate next step. Each workflow calls for different information and different permissions.
Dyna.Ai Voice Agent combines speech and conversation capabilities with configured business workflows. In a customer-service setting, Customer Service Agent can use approved knowledge to answer questions and route requests to a person when needed. Where account information or an action requires access to another system, the workflow needs the appropriate integration and authorization.
The distinction matters when something cannot be completed on the call. If an account lookup is unavailable, a polished voice cannot supply the missing balance. The customer needs to know what can be done next, and a receiving employee needs enough relevant context to continue. Successful handoff is part of the service, not an admission that the conversation failed.
A leading Singapore digital bank applied this approach to collections outreach. The bank needed to reach more customers while handling conversations across English, Singlish and Mandarin, with local accents and real-call conditions. Dyna.Ai Voice Agents connected to its telephony operation, handled repeatable first contact and captured conversation outcomes; bank employees continued to handle cases requiring judgment or follow-up.
The bank reported 3× collections efficiency and more than 1,000 outbound calls per hour. The example shows why speech capabilities matter: they form part of a workflow that has to reach customers, understand their responses and support the team responsible for what happens next.
Building voice AI for the way customers speak
A multilingual voice agent has to do more than recognize several languages. It needs to cope with the way those languages are used together, respond at the right moment and carry the customer's intent into the next step. Each part of that chain affects the outcome of the conversation.
Dyna.Ai brings multilingual speech, conversation management and voice generation into enterprise voice workflows, helping teams engage customers in their natural speech and move requests forward. Explore Dyna.Ai Voice Agent to see how it can support your customer conversations.