Voice AI
Production voice agents: the latency budget, what breaks, and how to scale (Part 3 of 3)
The final part of the build series. Part 1 made the case and the design decisions, Part 2 walked the code. This one is the post I wish had existed before I shipped: where the milliseconds go, what fails on real calls, and why the obvious scaling approach is wrong. Repo: github.com/zhang-liz/OpenLine. Video: the YouTube series.
TL;DR: The latency budget is won at end-of-turn detection, not the LLM. Four failure modes cover most real-world breakage. Scale out with capped workers, never threads in one process. And the fifth pillar almost everyone skips, observability, is where the compounding gains live.
A demo voice agent and a production voice agent are the same code with different enemies. The demo’s enemy is complexity. Production’s enemies are milliseconds, entropy, and Python’s GIL. This post covers all three, with the war stories, mine and the community’s, that taught me each lesson.
The latency budget: one second, itemized
Voice is the only AI modality where latency is not a quality metric, it is the quality metric. Here is roughly where a one-second voice-to-voice round trip goes:
Two things jump out of that chart. First, the biggest slice is not the AI thinking, it is deciding the human has finished talking. That is why Part 1 and Part 2 spent so much time on turn detection, and why the endpointing thresholds are the first thing to tune. An independent 2026 cost-and-latency calculator calibrated against production data puts default endpointing at roughly 1450ms on Vapi versus about 300ms on Pipecat, which is worth sitting with: the self-hosted pipeline is not just cheaper, its default turn-taking is faster than the platform you would be paying for. Second, no single stage dominates, which means no single upgrade saves you. The budget is won by streaming: every stage starts before the previous one finishes, so first audio reaches the caller while the model is still composing the rest of the sentence.
The thresholds that matter, and the same independent source lands on identical numbers: under about 1.5 seconds feels human. Between 1.5 and 2.5 is where most agents live. Past 3 seconds, callers assume the line dropped and say “hello?”, which barges in on your own agent and compounds the mess.
The tricks that hold this build under budget, in priority order: the opener is spoken directly through TTS with zero model wait (Part 2 showed the three lines), every stage streams, barge-in runs locally with no API round-trip, the system prompt is short, which matters more than people think, because the obvious alternative, prompt caching, has a gotcha almost nobody mentions: Anthropic’s cache only engages past a per-model minimum (2,048 tokens on Haiku-class models), so a lean voice prompt is often too small to cache, and short simply beats cached, and the agent is instructed to answer in one short sentence, because less text to generate and speak makes every single turn faster. Also prewarm on boot, and be precise about what that means here: the Silero VAD analyzer carries per-stream state, so it cannot be shared and is built fresh for every call. What can be paid once is onnxruntime’s lazy provider initialization and the model file’s first disk read, so the repo builds and discards one analyzer at startup. Skip that and your very first real lead eats the cold-start spike on their critical path.
Two wins the budget hides: cache the static prompt, and colocate the hops
Both are things I underplayed above, and both are close to free.
Cache the static prompt. I said a lean prompt beats caching, and for the instructions that is true. But a real qualifying agent also carries a big static block, the product facts, pricing, and policy the model answers from, and that block is re-sent on every single turn of every call. That is the case that clears Anthropic’s cache minimum with room to spare, and caching it turns the reference block from something you pay full freight for turn after turn into something you pay for once and read back at roughly a tenth of the input price, while trimming time-to-first-token because the model skips re-reading it. Two rules make the cache actually hit. Keep the static facts as a stable prefix and move anything per-lead to the very end, because a cache matches on an unchanged prefix, so a name spliced into the top invalidates everything after it. Here that is nearly done for you: the greeting already speaks the lead’s name, so the system prompt does not need it at all, drop the per-lead name from the prompt, make the whole thing static, and switch it on:
# prompt.py -- the system prompt now carries no per-lead data, so it never changes
def build_system_prompt() -> str:
return SYSTEM_PROMPT
# llm.py -- cache the static system prefix (Pipecat passes this through to Anthropic)
llm = AnthropicLLMService(
api_key=config.ANTHROPIC_API_KEY,
model=config.LLM_MODEL,
enable_prompt_caching=True,
)
The payoff scales with prompt size and call length: the longer your reference block and the more turns per call, the more of your LLM bill it erases. For a facts-heavy agent it is the single highest-leverage cost change in the build, and it costs one argument plus a prompt you were going to write carefully anyway.
Colocate the hops. The one-second budget quietly assumes the network is free; it is not. Every turn round-trips from your server to STT, to the LLM, and to TTS, and if those services sit in a different region from your box you pay cross-region latency, often 50 to 150ms, on each hop, several times per turn. Deploy your server in the region where most of your providers live, pick each provider’s nearest region where they offer the choice, and put the box near Twilio’s media edge. While you are there, warm the provider connections at boot the same way the repo warms Silero, so the first real call is not the one paying the TLS and HTTP/2 handshakes. None of this shows up in a demo of one local call; all of it shows up on the hundredth call from a datacenter.
What actually breaks: four failure modes
The community’s scar tissue is remarkably consistent. Across production write-ups, GitHub issues, and forum threads, the same four failures recur. One production guide for Pipecat documents 26+ real GitHub issues from teams running at scale; most fall into these buckets.
1. Turn-taking goes wrong. The signature failure of the whole category. Pipecat’s creator notes in his voice AI primer that the most common complaint about hand-rolled agents is interrupting too often; the opposite tuning leaves dead air. The Greylock team’s survey of production voice agents points at the deeper version: a caller’s “mhm” and “wait” are both interruptions, but only one should stop the agent, and plain voice-activity detection cannot tell them apart. Modern neural endpointing largely solves the basic case, which is why I delegated it to the STT provider, but budget real test time here. It is the difference between an agent people talk to and one they hang up on.
2. The audio turns to garbage. Phone lines carry narrowband 8kHz audio in telephony codecs. One sample-rate or codec mismatch anywhere in the pipeline and speech comes out garbled in one or both directions. This fails silently in the worst way: your code runs perfectly and the call is unusable. If your agent “cannot hear well,” check the audio format before you blame the STT model.
3. A provider stalls mid-call. STT, LLM, or TTS times out or rate-limits while a human is standing on the line. Text apps get a spinner; voice gets silence, and silence reads as a dead call. Every external call needs a timeout, and the agent needs a scripted fallback line and a graceful exit that still writes the transcript back. A half-finished call that leaves the lead’s row stuck in calling is a data bug on top of a UX bug.
Here is what “scripted fallback line” looks like as code, because it is the part everyone nods at and nobody builds. It is a tiny custom Pipecat processor that watches the gap between the caller finishing and the first agent audio, and if that gap blows past a threshold, speaks a holding line so the caller hears a human-shaped “one second” instead of dead air:
# watchdog.py -- a custom Pipecat processor, inserted just after the LLM
class StallWatchdog(FrameProcessor):
def __init__(self, timeout_s=3.0, line="Sorry, bear with me one second."):
super().__init__()
self._timeout_s, self._line, self._timer = timeout_s, line, None
async def process_frame(self, frame, direction):
await super().process_frame(frame, direction)
if isinstance(frame, UserStoppedSpeakingFrame): # caller done: start the clock
self._timer = asyncio.create_task(self._fallback())
elif isinstance(frame, TTSStartedFrame) and self._timer: # agent replied in time: all good
self._timer.cancel(); self._timer = None
await self.push_frame(frame, direction)
async def _fallback(self):
await asyncio.sleep(self._timeout_s)
await self.push_frame(TTSSpeakFrame(self._line), FrameDirection.DOWNSTREAM)
Drop it into the Pipeline list right after llm so it sees both the end-of-turn boundary and the TTS response, which is what lets it tell “still composing” apart from “stalled.” (Frame classes come from pipecat and track your version.) Two notes keep it honest. The holding line buys time, it does not fix the stall, so pair it with a hard per-request timeout on each provider call, so a wedged STT, LLM, or TTS actually errors out instead of hanging forever, and with the on_client_disconnected cancel the repo already has, so a stalled call still writes its transcript back. And keep the line generic: a neutral filler reads as a normal human pause, whereas “sorry, the system is slow” teaches the caller to distrust the agent for the rest of the call.
4. The socket drops and nothing notices. The media-stream websocket dies, the pipeline keeps trying to reconnect forever, and the write-back never fires. Part 2 covered the fix, explicit cancellation on disconnect, and I mention it again because it is the failure I see most in other people’s repos: the happy path works, the hang-up path was never tested. Test the hang-up path. Callers hang up mid-sentence constantly; it is the most common “edge” case there is.
Scaling: the intuition is wrong, here is the correct pattern
Here is the part where I correct something you will read in a dozen tutorials, and something I initially got wrong myself: “it is async Python, just add a worker pool and handle hundreds of calls in one process.”
No. Each call is a long-lived, stateful, CPU-touching session: continuous audio processing, VAD inference, streaming websockets. Python’s GIL means the CPU-bound parts of concurrent sessions fight each other, and the failure mode is documented and ugly. The production guide above reproduces it precisely: run concurrent pipelines in one process via a thread pool, and the first call works, the second freezes the whole service so hard that only SIGKILL recovers it.
The correct pattern, converged on independently by everyone running voice at volume, is to scale out, not up:
-
Cap sessions per worker. CallSphere’s write-up on scaling to 1000+ concurrent calls lands at 20 to 40 sessions per pod before event-loop contention bites. The strictest version is one container per session, which trades density for eliminating an entire class of concurrency bugs, and is exactly what managed hosts like Pipecat Cloud do under the hood.
-
Autoscale on active calls, not CPU. Voice load is bursty and session-shaped; CPU-based autoscaling under-provisions right when a calling campaign spikes.
-
Reuse connections. Recreating HTTP clients per call exhausts sockets under load. Pool them.
-
Drain gracefully. A deploy must let in-flight calls finish, not kill a live conversation mid-sentence. Wire drain to SIGTERM.
-
Back off before the provider does. Track your STT/LLM/TTS concurrency and apply back-pressure before you hit their rate limits, not after the 429s start.
For this lead-calling agent, the honest translation is: a single small instance comfortably covers the volume most lead pipelines actually produce, and when it does not, you add workers behind a router, not threads inside a process.
And this is where I will say something nice about the platforms I spent Part 1 undercutting: an isolated instance per call, autoscaled, drained on deploy, is precisely the service managed hosting sells for its per-minute fee. The break-even is knowable. Deploying your own pipeline to a managed host runs about a cent a minute; a small self-hosted box is roughly $25 a month fixed. The crossover sits near 2,500 minutes a month. Below it, managed hosting of your own pipeline is both cheaper and zero-ops. Above it, self-host and pocket the difference. Either way you are 4 to 5x under the managed orchestration platforms, because the fee you are avoiding was never the hosting, it was the orchestration.
Three guardrails that cost four lines
Before the observability sermon, three tiny changes that bound your worst-case cost and latency at the source. Each is a few lines; together they mean no single call can ever surprise you on the bill.
A carrier-side cost ceiling. Twilio will end a call at a hard limit for you, even if your server has wedged:
# twilio_client.py -- in calls.create(...)
time_limit=config.MAX_CALL_SECONDS, # default 300s: a runaway call cannot exist
This is the guardrail that survives every failure mode on your side, because it does not run on your side.
A dead-air guard. If nothing flows for two minutes, the caller walked away or you are streaming a voicemail’s silence at per-minute rates. Cancel and let the write-back fire:
# pipeline.py -- on the PipelineTask
idle_timeout_secs=120,
A per-turn token cap. The prompt demands one short sentence; this makes a runaway monologue impossible at the API level, bounding both per-turn latency and per-turn spend:
# llm.py
params=AnthropicLLMService.InputParams(max_tokens=120),
The shape to notice: each guardrail lives at a different layer (carrier, pipeline, model), so no single bug can defeat all three. That layering instinct is most of what “production-grade” means.
The fifth pillar: observability and evals
Every stack diagram shows four pillars: hear, think, speak, connect. The teams running voice at real volume all converge on a fifth that the diagrams omit: knowing whether any of it is actually working. Hamming’s analysis of 4M+ production calls calls monitoring the part everyone forgets, and it matches what I see in every build thread: weeks on the pipeline, zero minutes on measurement.
The good news for this architecture is that the raw material is already flowing. Transcripts land in the sheet on every call, and Pipecat’s built-in metrics (already enabled in this repo’s PipelineParams) emit per-stage timings, so you can decompose latency into end-of-turn delay, LLM time-to-first-token, and TTS time-to-first-audio instead of guessing which stage regressed. From there, the minimum viable eval loop is one scheduled job: run an LLM-as-judge pass over the day’s transcripts scoring three binary questions, did the agent qualify correctly, did it handle the main objection, did the caller hang up mid-agent-sentence. That last one is your turn-taking regression alarm, and it is the single highest-signal metric in the whole system, because every upstream failure, bad endpointing thresholds, a slow model swap, a garbled codec, eventually shows up as humans hanging up on a robot that talks over them.
Two habits to steal from teams doing this well: evaluate continuously rather than at release time, because vendors update models under stable version strings and your agent’s behavior drifts without a single line of your code changing; and keep a small fixed set of recorded test calls you replay after every prompt or provider change, which turns “did the swap break anything” from a quarter-long bake-off into an overnight regression diff.
What I would build next
Where this system goes from here, in the order I would ship it:
Voicemail detection is the biggest gap for outbound specifically. A meaningful share of outbound calls hit voicemail, and an agent that pitches a beep is both wasteful and embarrassing. Retell’s engineering write-up on the hardest moments in real calls puts answering-machine detection in the top four behaviors separating demos from production, and I agree.
The good news is Twilio does the hard part; you enable answering-machine detection when you place the call and branch on the verdict it posts back:
# twilio_client.py -- add to _client.calls.create(...) in place_call()
machine_detection="Enable", # Twilio classifies human vs machine
async_amd="true", # classify in the background, do not delay a human pickup
async_amd_status_callback=f"https://{config.PUBLIC_URL}/amd?row={row_number}",
# app.py -- new /amd endpoint, a sibling of the existing /status handler
@app.post("/amd")
async def amd(request: Request, row: int):
form = await request.form()
answered_by = form.get("AnsweredBy", "") # human | machine_start | machine_end_beep | fax | unknown
if answered_by.startswith("machine"):
sheet.write_result(row, "voicemail", f"amd: {answered_by}") # log and stop; or leave a short VM
twilio_client.end_call(form.get("CallSid", "")) # hang up on the machine
return PlainTextResponse("", status_code=204)
That end_call is a two-line helper you add next to place_call: _client.calls(sid).update(status=“completed”). Two honest caveats before you turn it on. Detection is not instant: AMD listens for a beat before releasing the human path, so it trades a slice of latency against never pitching a beep, worth it for outbound, rarely for inbound. And it is probabilistic, so it will occasionally tag a slow-answering human as a machine; keep the misclassification rate in the eval loop from the section above and tune machine_detection_timeout rather than trusting the default. This is the single biggest outbound win on this list, and the reason it is a “next” rather than a baseline feature is that it genuinely changes the call flow and adds a webhook path, which is more than belongs in a snippet trimmed for clarity.
Retry the misses. Straight after voicemail, because they are the same problem: on outbound a large share of first dials go to no-answer or machine, and a single attempt leaves most of your list uncontacted. Add an attempts column, and have the /status and /amd handlers increment it and re-queue the row for a later dial instead of marking a miss final, two or three tries spaced over the next hour, then give up. The sheet is already the queue, so this is a counter and a timestamp check, not new infrastructure, and it is often a bigger contact-rate lift than any model or voice choice, because the cheapest way to talk to more leads is to call back the ones who did not pick up.
Human handoff. When a lead is hot, or asks for a person, the right move is a warm transfer to a rep, not an AI trying to close. The webrtc.ventures team’s policy-guardrails architecture is the pattern I would follow for anything where the agent can take consequential actions: the model interprets, deterministic code decides.
In-call disclosure. Consent-to-call is built in, and Part 2 now folds a recording-and-AI disclosure straight into the GREETING constant. The layer beyond that is making the wording jurisdiction-aware, since what you must disclose, and how, varies by region, so drive it from config keyed to the lead’s location rather than shipping one hard-coded line. Treat it as product design, not legal fine print: leads respond better to an agent that is upfront.
Post-call structured extraction. The transcript is the record, but a transcript is not a CRM field. High-value, low-code: after write_result() persists the transcript, run one more cheap pass over it that returns strict JSON, qualified yes/no, timeline, budget, the main objection, and write each value into its own column on the same row.
# extract.py -- one off-hot-path pass, run after the transcript is written back
from anthropic import Anthropic
SCHEMA = '{"qualified": bool, "timeline": str, "budget": str, "objection": str}'
client = Anthropic(api_key=config.ANTHROPIC_API_KEY)
def extract(transcript: str) -> dict:
msg = client.messages.create(model=config.LLM_MODEL, max_tokens=200,
messages=[{"role": "user", "content": f"Return ONLY JSON matching {SCHEMA} for:\n{transcript}"}])
return json.loads(msg.content[0].text) # parse defensively: log and skip on bad JSON
Three reasons it earns a slot. It runs off the hot path, so it uses a plain Anthropic call rather than the streaming make_llm() service, and it is the one place in the whole build where you should reach for a smarter model if it extracts cleaner, call latency is untouched, so point LLM_MODEL at a bigger model just for this pass. It reuses the key and the sheet you already have, then drops the four fields into new columns with a couple more update_cell calls. And it turns a call log into a pipeline: “show me every qualified lead with a Q3 timeline” becomes a sort, not a listening session. Keep the prompt strict about JSON-only output and parse defensively, because a model that editorializes instead of returning JSON should degrade to a blank column, not a crashed job.
Number reputation. Outbound numbers get flagged as “Spam Likely,” and a flagged number silently destroys pickup rates in a way no per-minute saving compensates for. Register your traffic, use branded caller ID where available, and monitor flagging from day one.
The series in three sentences
Speed wins leads, and an event-driven trigger plus a streaming pipeline is how you get it. The managed platforms charge a nickel a minute for an orchestration layer that is, in the open-source version, a Python list you can read. And the gap between a demo and production is not more AI, it is turn-taking, timeouts, hang-up handling, and scaling out instead of up.
The code is open: github.com/zhang-liz/OpenLine. The video walkthrough is here: the YouTube series. Build it, call yourself first, and tell me what broke, because that is the part of this field where all the interesting work is.