- The classic speech-to-text then LLM then text-to-speech pipeline stacks latency at every stage and flattens prosody; realtime and speech-native models cut end-to-end delay toward conversational range and keep tone.
- Conversation is a latency product, not just an accuracy product: natural turn-taking wants responses on the order of a few hundred milliseconds, and every network, model, and audio hop spends against that budget.
- The contact center is the flagship use case and the reason the governance bar rises: errors are spoken aloud instantly, barge-in and endpointing are hard, and graceful human handoff is mandatory.
Why voice is suddenly viable
Enterprises have run automated voice for decades, and almost everyone has hated it. The interactive voice response tree, the "please say or press one," the flat synthetic readback: these were not conversations, they were menus wearing a voice. What changed is not that models learned to talk. It is that they learned to talk back fast enough, and with enough of the original tone intact, that the exchange starts to feel like one between two people rather than a query submitted to a machine.
The technical reason is worth stating plainly, because it reframes the whole category. For years the only way to build a voice agent was to chain three separate systems: transcribe the speech to text, feed that text to a language model, then synthesize the reply back into audio. That chain works, and it powered the first wave of usable voice bots. But it stacks delay at every joint, and it throws away everything about how something was said, keeping only the words. A caller who is frustrated, hesitant, or joking sounds identical to the model once their speech is flattened into a transcript.
Two shifts moved voice from tolerable to genuinely conversational. Streaming realtime interfaces let audio flow in and responses flow out continuously instead of in discrete request-response turns. And speech-native models, which reason over audio directly rather than routing through a text bottleneck, cut the end-to-end delay toward the range human conversation expects while preserving prosody, emotion, and timing. Major providers shipped realtime voice APIs across 2024 and 2025; the specifics are moving quickly and worth treating as a direction rather than a fixed set of products.
The cascaded pipeline and its ceiling
The cascaded pipeline is the honest baseline, and understanding exactly where its ceiling sits explains why realtime models mattered. Three stages run in sequence: speech-to-text (STT) turns the caller's audio into a transcript, the language model reads that transcript and produces a reply, and text-to-speech (TTS) turns the reply back into audio. Each stage is a mature, capable technology on its own. The problem is not any single stage; it is that they are serial and that the boundaries between them are hard.
The hardest boundary is the first one. The language model cannot begin reasoning until it has a transcript, and it cannot have a confident transcript until the caller has actually stopped talking and the STT engine has committed its output. So the model's clock does not even start until the human's turn is fully over and recognized. Then generation runs, and only after the reply text exists can synthesis begin producing sound. The caller hears nothing until that last stage yields its first audio.
Caller speaks ......................] (turn ends)
|
[ STT ] transcribe + endpoint ##
|
[ LLM ] read transcript, generate ######
|
[ TTS ] synthesize reply audio ####
|
Caller hears first sound ^
|------------------ perceived silence ------------|
The costs compound rather than add in the way that matters to feel. It is not only the raw milliseconds; it is that the silence lands precisely where a human listener expects an immediate reaction, and its length is unpredictable because it depends on utterance length and model load. Engineering can shave the total, streaming partial transcripts, starting synthesis on the first sentence, but the serial dependency is structural. You can make the cascade faster; you cannot make it feel simultaneous, because by design nothing downstream can start until something upstream finishes.
Realtime and speech-to-speech models
Realtime models attack the structural problem rather than shaving the stages. A streaming realtime API accepts audio as it arrives and emits audio as it is ready, so the neat serial boundaries of the cascade dissolve into a continuous exchange. The model can be forming a response while the caller is still finishing, and it can begin speaking without waiting for a fully committed transcript. The interaction stops being a sequence of complete turns and becomes an overlapping duplex stream, which is how human conversation actually works.
Speech-native, or speech-to-speech, models go a step further by reasoning over audio directly instead of collapsing it to text and back. Because the text bottleneck is removed, the information that the cascade discarded, how something was said, survives into the model's understanding and out into its reply. That is what lets a response match the caller's pace, soften when they sound distressed, or carry the timing cues that make an exchange feel alive. The trade is maturity and control: a text intermediate is inspectable, loggable, and easy to guardrail, while a pure audio path is harder to observe. Many production designs therefore keep a text transcript alongside the audio path for exactly those reasons.
| Property | Cascaded STT→LLM→TTS | Realtime / speech-native |
|---|---|---|
| Processing shape | Serial, turn by turn | Streaming, overlapping duplex |
| Prosody and emotion | Largely lost at the transcript | Preserved end to end |
| End-to-end latency | Sum of three stages | Toward conversational range |
| Inspectability | High: text at every seam | Lower without an added transcript |
| Maturity | Well understood, stable | Newer, evolving quickly |
Treat the table as a snapshot, not a verdict. The realtime and speech-native category is moving fast enough that specific capabilities, latencies, and controls shift release to release. The durable point is architectural: the cascade optimizes for inspectability at the cost of feel, and the realtime path optimizes for feel at the current cost of some observability. Where an organization lands depends on how much of each it can afford on a given channel.
The latency budget conversation demands
The single most important reframing for anyone building voice is this: in a conversation, latency is the product. A voice agent can be more accurate, more knowledgeable, and better governed than any human agent, and still feel broken if its replies land half a second too late. Human turn-taking runs on a tight clock; the comfortable gap between one person finishing and the other starting is short, on the order of a few hundred milliseconds, and listeners register anything much longer as hesitation, confusion, or a dropped call.
That number is not a target you design toward at the end. It is a fixed budget you allocate at the start, and every component in the path spends against it. The network round trip between caller and infrastructure takes a slice. Detecting that the caller has finished takes a slice. The model's own processing takes the largest and most variable slice. Audio buffering on the way out takes another. What is left after the fixed costs is what the model actually gets to think in, and it is far less than the headline budget suggests.
| Where the budget goes | Nature of the cost |
|---|---|
| Network transport, both directions | Fixed by geography and routing; shrink with edge presence |
| Endpointing / turn detection | Tunable, but trades speed against false cutoffs |
| Model inference | Largest and most variable; the core engineering target |
| Audio encode, buffer, playout | Small but real; adds jitter under load |
The numbers above are deliberately left as shapes rather than figures, because they depend on region, model, and load and because quoting exact milliseconds as fact would mislead. The discipline is what transfers: budget latency the way you budget cost, assign every hop an allowance, and measure the tail rather than the average. A median that feels responsive with a long tail that occasionally stalls for two seconds is not a good system; it is a good system with an intolerable failure mode that callers will remember.
Interruption, turn-taking and barge-in
Text conversations are strictly turn-based: one party sends, the other reads, nobody talks over anybody. Voice is nothing like that, and the difference is the source of most of the hard engineering. People interrupt. They cut in with a correction, an "actually, no," or an answer to a question before it finishes. A voice agent that cannot handle being interrupted feels less like a conversation partner and more like a recording that refuses to stop, which is exactly the failure the whole category is trying to escape.
Barge-in is the capability of detecting that the caller has started speaking while the agent is still talking, stopping its own output immediately, and switching to listening. It sounds simple and is not. The system has to distinguish genuine interruption from background noise, from the caller's own "mm-hmm" backchannel, and from the agent hearing its own voice through the line. Getting it wrong in either direction is jarring: an agent that talks over a caller is rude, and one that stops at every cough is skittish.
- Endpointing: deciding when the caller has actually finished, not merely paused. Cut too early and you interrupt them mid-thought; wait too long and you burn the latency budget on silence.
- Barge-in detection: recognizing real speech over the agent's own output and yielding the floor fast, without tripping on noise or backchannel sounds.
- Echo handling: preventing the agent from mistaking its own synthesized voice, returning through the audio path, for the caller speaking.
- Backchannel awareness: treating brief "yeah" and "okay" cues as encouragement to continue rather than as a request to stop.
Endpointing deserves special weight because it sits on the critical path of feel. Humans signal the end of a turn with a mix of grammar, intonation, and rhythm, not a clean pause, and a system that waits for a fixed silence threshold will always be either too eager or too slow. This is one of the genuinely open problems in the space, and it is where realtime models that reason over audio timing, rather than over a silence timer, have the most to offer.
The enterprise stakes: the contact center
The reason voice attracts serious enterprise investment is not novelty. It is that voice agents point directly at the contact center, which in many large organizations is the single largest human-operated function, measured in headcount, in cost, and in the sheer number of daily customer interactions it owns. Automating even a meaningful fraction of that volume changes the economics of a whole business unit, and unlike a text chatbot, a competent voice agent can address the calls that customers still, overwhelmingly, prefer to make by phone.
The size of the prize is matched by the sharpness of the risk, and the two are inseparable. In a text channel a wrong answer sits on a screen where it can be caught, edited, or quietly retracted. In a voice channel a wrong answer is spoken aloud, in real time, in a confident synthetic voice, directly into a customer's ear, and it cannot be unsaid. There is no draft state, no review step, no undo. Everything the agent produces is published the instant it is generated, to an audience of one who is often already unhappy.
Two obligations follow and are not optional. The first is escalation: the agent must recognize the situations it should not handle, distress, complex disputes, anything with legal or financial weight, and route them onward rather than improvising. The second is graceful human handoff: when it escalates, it must transfer the caller to a person with the full context of the conversation so far, not dump them back into a queue to start over. A handoff that loses context is worse than no automation at all, because it spends the customer's patience twice.
What architects must design for
Voice changes the architect's job in ways text never forced. The obvious shift is that latency graduates from a performance metric into a contractual one. On a voice channel the response-time budget is a hard service-level objective, and the tail of the distribution matters more than the median, because callers forgive a system that is consistently quick far sooner than one that is usually quick and occasionally stalls. Design the whole path, network, endpointing, inference, audio, to hold a p95 you can defend, not an average you can advertise.
The second shift is that streaming must run end to end. Any component that waits to receive a complete input before producing a complete output reintroduces the cascade's stall, so batch-shaped thinking has to be pushed out of the entire path. And because the channel publishes instantly, guardrails have to operate as fast, low-latency filters that run inside the budget rather than as a leisurely post-generation review; on a channel with no draft state, a slow guardrail is no guardrail. The threat model widens too, since prompt-injection and manipulation attempts now arrive as spoken audio.
- Latency as an SLA. Set explicit budgets per hop, measure the tail, alarm on p95 regressions, and treat a latency breach as an incident, not a tuning note. See inference optimization for where the model's slice can be reclaimed.
- Streaming everywhere. No component may block on a complete input or emit only a complete output; partial in, partial out, all the way through.
- Graceful human handoff. Build escalation and warm transfer that carries full conversation context to the human agent, as a first-class flow rather than an afterthought.
- In-budget guardrails. Enforce PII, safety, and injection defenses as fast filters on the live path. The prompt-injection defense and privacy and PII playbooks apply, now against spoken input.
The uncomfortable summary for a skeptical CTO is that voice does not relax any of the governance you already owe on text; it applies all of it, faster, with the mistakes read aloud. That is not a reason to avoid the channel, given the size of the contact-center prize. It is a reason to treat voice as the most demanding place to prove your controls actually work under a clock, and to build there deliberately rather than by porting a text agent and hoping. Prototype the hard parts, endpointing, barge-in, handoff, in a sandbox like the Agent Studio in the Lab before they meet a live caller.