Voice AI Latency Budgets: Serve Speech Close to the Speaker

Text interfaces tolerate delay. Voice does not. Across a sample of ten languages, Stivers et al. found a general avoidance of overlapping talk and a minimisation of silence between turns. Levinson and Torreira put the typical gap between turns at around 200 milliseconds, even though producing speech takes people over 600 milliseconds, which means listeners plan replies before the other person stops talking.

A voice agent has no such head start unless it is engineered for one. This article lays out a latency budget and the placement decisions that follow from it.

The budget

Telephony gives a long-standing reference point. ITU-T G.114 treats one-way delay below 150 milliseconds as essentially transparent for most applications, and recommends not exceeding 400 milliseconds one-way for network planning. A voice agent adds computation on top of transport. A realistic end-to-end budget for a reply that feels natural splits roughly like this:

Each stage that crosses a long network path pays the distance twice. Light in optical fibre covers roughly 200 kilometres per millisecond, so a server a continent away adds tens of milliseconds per round trip before any computation. Several such hops consume most of the budget.

The takeaway: count round trips, not just compute. Every stage placed far from the user spends budget on transport.

Where each model should run

Speech recognition: near the user. Recognition sees a continuous stream and must finish quickly after the user stops. Models designed for live use help: Moonshine reports that its tiny model needs about five times less compute than Whisper tiny.en to transcribe a 10-second segment, with no increase in word error rate on standard test sets, by processing variable-length audio without padding. Small, fast recognisers fit on modest hardware in a metro or carrier edge site.

Speech synthesis: near the user. The first audio chunk gates perceived latency. Streaming synthesis that starts speaking after the first phrase, served close to the listener, keeps that chunk early.

Language model: wherever quality and price are right, with streaming. The language model often needs the most capable hardware and can tolerate more distance, provided it streams tokens so synthesis can start on the first phrase. Keep conversation state warm on the serving machine so each turn only processes the new words.

Turn detection: as early as possible. Waiting for a fixed silence wastes budget. Partial transcripts let the agent begin retrieving context and even drafting a reply before the user finishes, much as people do.

Engineering against the slow tail

Average latency hides the problem. In voice, the occasional slow reply is what users remember. Dean and Barroso showed that in large distributed systems, tail latency dominates user experience and can be reduced by hedging: sending a second copy of a slow request to another server and taking whichever answers first. For voice:

• Hedge recognition and first-token requests when the primary is slower than its own recent 95th percentile.
• Detect failed servers in seconds, not minutes, and fail over mid-conversation without losing context.
• Keep a warm standby that already holds the conversation's cache, so a failover does not restart from scratch.

Regional notes

Voice demand is global, and many fast-growing markets in Africa, Latin America, MENA and South and Southeast Asia have fewer large data centres nearby, which makes long round trips to distant regions the default. Serving recognition and synthesis from capacity inside the region (metro data centres, carrier edge sites, even on-premises servers) matters more there than anywhere else. Voice recordings are also often personal data: in the EU and UK they fall under GDPR, and biometric uses of voice carry stricter rules in many jurisdictions.

What this means for developers and telcos

• Write the latency budget down per stage and measure each one in production.
• Put recognition and synthesis in the same metro as users; let the language model stream from wherever it is best.
• Engineer for the 95th and 99th percentile, with hedging and fast failover.
• For telcos: speech models are small enough to run at the network edge, and voice agents are latency-sensitive enough to value it.

Where Tenzro fits

Tenzro is an open network where machines of every size serve AI to agents, apps and people. For voice, built today:

• Speech models. Providers serve recognition (Moonshine v2, Distil-Whisper, Whisper-large-v3-turbo, Parakeet-TDT and Canary-1B-Flash) and speech synthesis (Qwen3-TTS, also via an OpenAI-style speech route).
• Realtime surfaces. A realtime turn endpoint takes one spoken turn and streams back the transcript, reply text and audio chunks over server-sent events; streaming transcription returns partial and final transcripts, with each final segment billed by the second and covered by a signed receipt.
• Placement. Providers declare a coarse region and jurisdiction, and live load per model (warm or cold, in-flight requests, queue depth), so an application can choose a nearby, warm provider.
• Tail control. The router hedges a request to a second provider when the primary exceeds its own observed 95th-percentile latency (clamped between 40 and 500 milliseconds by default), and treats a provider as down after six seconds without a signed heartbeat.
• Session continuity. A multi-turn chat session keeps its KV cache across turns and can name a successor in a permitted jurisdiction that receives sealed cache checkpoints, so a failover continues without re-processing the conversation.

The edge build for small devices forwards model calls and does not run realtime speech itself. The network is live in preview. See tenzro.com.