Realtime Voice Agents

Voice AI Agent Development

Conversations live or die in the 800 milliseconds after the caller stops speaking

Every voice agent sounds good in a demo. On a real phone call the caller gets talked over, cut off mid-sentence, or left listening to silence while a tool call resolves. We build voice agents on LiveKit, Whisper and ElevenLabs the other way round — latency budget first, endpointing tuned to the channel, barge-in handled properly.

LIVEKITWHISPERELEVENLABSENDPOINTINGBARGE-INSIP / PSTNWEBRTC

Sub-second

Target end-to-end response budget, measured stage by stage

Barge-in

Interruption handling designed in, not patched on afterwards

SIP + WebRTC

The same agent on a phone line and in the browser

Replayable

Tuned against recorded calls, not live experiments on customers

Why Zenthos

Demos sound great. Production calls are where voice agents fall apart.

In a demo you speak clearly, one sentence at a time, into a laptop microphone in a quiet room, over WebRTC, watching a UI that shows the agent is thinking. In production the caller is on a mobile in a moving car, over a narrowband phone codec, with no visual feedback whatsoever. The only evidence that anything is happening is whether they can hear a voice. A 300ms gap that feels instant in a browser reads as dead air on a phone line — and somewhere past a second, people start saying “hello?” and talking over the answer that was about to arrive.

The failure modes are specific and they compound. Endpointing fires too early and the agent confidently answers a sentence the caller had not finished. Raise the silence threshold to fix that and every single turn now costs an extra few hundred milliseconds. Barge-in that is not handled properly means the agent keeps talking over an interruption — or stops, but still believes it said every word it generated, so the rest of the conversation is built on sentences the caller never heard. And each tool call you add, a CRM lookup or an availability check, is a full extra round trip through the model. That is usually where a sub-second agent quietly becomes a three-second one.

So we start with a latency budget rather than a prompt. Where the milliseconds actually go — VAD and end-of-turn detection, ASR finalisation, LLM time-to-first-token, TTS time-to-first-byte, and the network path underneath all of it — then we design against that budget: streaming at every stage, slow lookups moved off the critical path, unavoidable waits covered with speech instead of silence, endpointing tuned per channel. And we tune on recorded calls with real accents, real background noise and real interruptions, because a voice agent tested only by the team that built it has been tested exclusively by people who already know what to say.

What We Deliver

Core Capabilities

[01]

Latency Budget Design

We instrument every stage — VAD, ASR finalisation, LLM time-to-first-token, TTS time-to-first-byte, transport — and allocate a millisecond budget to each before the agent is written. When a call feels slow you then know which stage to attack, instead of swapping models and hoping.

[02]

ASR & Endpointing Tuning

Streaming transcription with Whisper, Deepgram or whichever model suits your audio, plus end-of-turn detection tuned per channel. Endpointing is the single biggest lever on perceived responsiveness, and the silence threshold that feels natural in a browser is the wrong one for a phone line.

[03]

Barge-In & Turn-Taking

Interruptions stop playback within a couple of hundred milliseconds, cancel in-flight generation, and — the part most implementations miss — truncate conversation state to what the caller actually heard. Plus echo cancellation, so the agent does not interrupt itself.

[04]

Telephony & SIP Integration

Inbound and outbound calling over SIP trunks and the PSTN, through LiveKit SIP, Twilio or the carrier you already use. DTMF capture, warm transfer, voicemail detection, recording, and the narrowband codec that changes how your ASR behaves.

[05]

Tool Calling Mid-Conversation

Account lookups, availability checks and order status without dead air: prefetch in parallel with the greeting, cache what is cacheable, speculatively execute what is safe to execute, and speak honestly while the rest resolves. Every tool call is a round trip and gets budgeted as one.

[06]

Evaluation on Recorded Calls

A regression suite of real audio — accents, noise, interruptions, callers who trail off mid-sentence — replayed against every change, scoring transcription accuracy, turn-taking behaviour and per-stage latency. You change a prompt and find out what broke before a caller does.

How It Compares

Voice agent stacks compared

Four realistic ways to ship a voice agent. The honest difference between them is how much of the pipeline you own — and therefore how much of it you are responsible for.

LiveKit AgentsPipecatOpenAI Realtime APIVapi
Control over pipelineHigh — you write the agent loop, plugins per stageHighest — explicit frame pipeline you assemble yourselfLow — a speech-to-speech model handles turns internallyConfiguration-level — you tune settings, not the media path
Swap ASR or TTS independentlyYes — provider plugins, swappable per stageYes — vendor-neutral services by designNo — audio in, audio out from one model (voice choice only)Yes, from the providers the platform supports
Telephony supportSIP bridging for inbound and outbound callsThrough transports — Twilio, Telnyx, Daily and similarNot the product’s centre of gravity; you bridge the carrier legIncluded — numbers, routing and carrier handled for you
Self-hostableYes — open-source server, or their cloudYes — runs anywhere, or their cloudNo — hosted APINo — managed platform
Time to first prototypeA day or twoDays — you wire the pipelineHoursUnder an afternoon in the dashboard
Best whenYou need production WebRTC and telephony under your own controlYou want to own every stage of the media pipelineNatural speech-to-speech matters more than component choiceA fairly standard call flow needs to be live this week

If your use case is a scripted flow — appointment reminders, qualifying inbound leads, order status — a managed platform is genuinely the right answer, and cheaper than what we would charge to rebuild it. Build custom when the latency budget is tight, when audio or transcripts cannot leave a jurisdiction, when the telephony routing is unusual, or when the agent needs to reach deep into systems a platform will never integrate with.

How We Work

Our Process

01

Call Flow & Latency Budget

We map the conversations the agent actually has to hold, then write the millisecond budget per stage before picking any model. This is also where we decide what must happen during the call and what can happen after it — the cheapest latency win is work you move off the call entirely.

02

Stack Selection

Cascading ASR → LLM → TTS or a speech-to-speech model, self-hosted or managed, chosen against the budget and the control you need. If the honest answer is that a platform covers your case, we say so at this step rather than three months in.

03

Build & Instrument

The pipeline on LiveKit Agents or Pipecat, streaming at every stage, with per-stage timing traces from the first commit. Barge-in, interruption handling and conversation-state truncation go in now — retrofitting turn-taking into a working agent is close to a rewrite.

04

Tune Against Real Audio

Endpointing thresholds, ASR hints for names, products and codes, TTS pacing and filler behaviour — all tuned on recordings of your actual callers on your actual channel. Accented, noisy and interrupted audio is the test set, not the edge case.

05

Deploy, Monitor & Hand Over

Containerised into your infrastructure, with per-call traces and latency percentiles in your observability stack, recordings and transcripts wherever compliance needs them, a runbook, and the evaluation suite in CI. You own the code.

Tools & Technologies

Tech Stack

We choose the right tool for the job — not the trendiest one.

LiveKit
Pipecat
Python logoPython
Whisper
Deepgram
ElevenLabs
OpenAI Realtime logoOpenAI Realtime
Twilio
WebRTC
Questions

Frequently Asked Questions

What voice agent latency is actually achievable?
On a well-tuned cascading stack, a sub-second gap between the caller finishing and hearing the first syllable back is a realistic target, and speech-to-speech models can go lower. But the raw number matters less than where it comes from. End-of-turn detection alone often accounts for several hundred milliseconds, ASR finalisation and LLM time-to-first-token a couple of hundred each, TTS time-to-first-byte another hundred or two, and the network path the rest. Anyone quoting you a single figure without saying which stages it covers, on which channel, is quoting a demo.
Why does our voice agent keep interrupting callers?
Almost always endpointing. The agent is deciding the caller has finished based on a fixed silence threshold, and people pause mid-sentence — to think, to read a number off a card, to breathe. Shortening the threshold makes the agent faster and ruder; lengthening it makes it polite and sluggish. The fix is not one number: it is semantic turn detection that considers whether the sentence sounded finished, thresholds tuned per channel, and correct handling of the case where the caller starts speaking again while the agent is already talking.
Can the agent handle real phone calls over SIP or PSTN?
Yes, and it is a materially different problem from a browser call. Phone audio arrives narrowband and often transcoded, which measurably hurts transcription of names, postcodes and reference numbers. There is no visual channel, so silence is your only feedback mechanism. And you inherit carrier behaviour: jitter, DTMF, voicemail systems that answer instead of humans. We connect through LiveKit SIP, Twilio or your existing trunk, and we tune for the channel rather than porting the browser configuration onto a phone line.
How do you handle strong accents and background noise?
Three levers, in order of impact. First, choose the ASR model on your audio rather than on a benchmark — accuracy on accented, narrowband, noisy speech varies far more between models than headline word error rates suggest. Second, bias the recogniser with your actual vocabulary: product names, place names, SKUs, spelled-out reference codes. Third, design the conversation to be robust — confirm critical values back, offer DTMF entry for numbers, and let the agent ask rather than guess. Then measure it on recordings from the callers you actually get.
What does a voice agent cost per minute at scale?
Four separate meters run at once: ASR per minute of audio, LLM tokens per turn, TTS per character or second generated, and telephony per minute. Single-digit US cents per minute is a reasonable planning figure for a cascading stack before carrier costs, but the spread is wide. The line item people underestimate is the LLM, because voice agents resend the whole conversation on every turn — so cost grows with call length, and prompt size is usually the biggest lever you control. Provider pricing moves fast enough that any number printed on a web page is wrong within a quarter, so we build a per-minute model with current rates during scoping.
Can it transfer the call to a human?
Yes — cold transfer, or a warm transfer where the agent briefs the human first. What matters more than the mechanism is the policy: which triggers escalate (explicit request, repeated misunderstanding, a high-value or regulated intent, detected frustration), what the human receives on pickup (a transcript and a summary, not a cold start), and what happens when no one is available. That last path is the one most implementations skip, and it is the one your callers will find.
How do you test a voice agent?
A regression suite of recorded calls, replayed through the pipeline on every change. Each run scores transcription accuracy against a reference, whether the agent completed the task, turn-taking behaviour — false interruptions and missed endpoints — and per-stage latency at p50 and p95, not just the average. We add adversarial audio deliberately: interruptions, cross-talk, callers who trail off, hold music, voicemail greetings. Manual dialling by the build team is not a test suite; it is a group of people who already know the happy path.
Do you work with companies in the US, UK and EU?
Most of our engagements are with teams in those markets. We work overlapping hours with both US coasts and European time zones, and we are used to operating inside client security reviews, DPAs and procurement. Voice adds its own compliance surface — call recording consent rules differ by jurisdiction and by US state, and audio plus transcripts are often subject to residency requirements — so we treat recording, retention and data location as design decisions at the start rather than questions for legal at the end.
Insights

Related Articles

View all →

Ready to build a voice agent that survives a real phone call?

Send us a recording of a call your agent got wrong, or just describe the flow you need. We will tell you where the milliseconds are going — and whether you should build this or buy a managed platform.