Liz Zhang

Voice AI

I built an AI agent that calls my leads in seconds, for about 80% less. Here is the full design. (Part 1 of 3)

By Liz Zhang · · 10 min read

← Back to blog

Part 1 of a three-part build series. This post covers the why and the design decisions. Part 2 walks through the code, Part 3 covers latency, failure modes, and scaling. If you prefer video, the full build is on YouTube: the YouTube series, and the code is open source: github.com/zhang-liz/OpenLine.

TL;DR: An event-driven trigger plus a streaming Pipecat pipeline calls leads within seconds of a form submission, at $0.05 to $0.06 per minute all-in versus $0.15 to $0.30 on managed platforms. The savings are not cheaper parts, they are a skipped orchestration fee.

By the end of this series you will have a voice agent that calls a lead within seconds of them submitting a form, holds a natural qualifying conversation, and writes the transcript back to a Google Sheet. It runs on plain Python you can read top to bottom, and it costs about $0.05 per minute all-in, against the $0.15 to $0.30 per minute a typical managed stack lands at.

That gap is not because the parts are cheaper. It is because most builds pay a tax they do not need to pay. This post explains where that tax hides, and walks through every design decision with the alternatives I considered, so you can disagree with my picks intelligently.

The two numbers that justify the whole project

Before any architecture, the business case, because it decides the architecture.

Number one: 21x. A lead contacted within 5 minutes is about 21 times more likely to qualify than one contacted at 30 minutes. That is from Dr. James Oldroyd’s 2007 MIT / InsideSales lead response study, and yes, it is from 2007. I cite the year deliberately, because the second number proves nothing has changed since.

Number two: 7%. A Drift audit found only 7% of companies respond to a lead within 5 minutes. A 2024 Workato study of 114 companies found exactly zero called within that window. Harvard Business Review’s audit of 2,241 firms measured an average response time of about 42 hours, and found firms responding within an hour were nearly seven times more likely to qualify the lead.

Put together: the window where a lead converts best is sitting wide open because almost nobody is fast enough to use it. And a widely circulated Lead Connect survey found 78% of buyers go with whoever responds first. Not the best vendor. The first one.

So the entire job of this system is to be the first responder, every time, automatically. Every design decision below is downstream of that goal plus one constraint: keep it cheap enough to run on real volume.

The architecture: one closed loop

A lead submits a form (your site, or a Google Ads lead form). A row lands in a Google Sheet. An event-driven trigger posts that row to a small FastAPI server, which checks consent and dials immediately. The agent qualifies the lead, and when the call ends, the status and full transcript write back to the same row. No dashboard, no queue service, no polling.

Two details in that loop carry more weight than they look like they do.

The trigger is event-driven, not polled. A Google Apps Script onChange fires inside the Sheet the instant a row is written and POSTs the row number to the server. The common no-code pattern polls the sheet on a timer, checking every few minutes, and that lag alone can eat the entire 5-minute window the 21x number lives in. Event-driven is not just cleaner engineering, it is the mechanism that makes “calls in seconds” physically true.

Consent is checked before every dial. Rows carry a consent flag captured at the form, and the server refuses to dial without it. In the US, automated and AI-voice calls without prior express consent are a TCPA problem, and the FCC has tightened AI-voice rules specifically. Designing the consent gate into the trigger path, rather than bolting it on later, is the difference between a lead-response system and a robocall machine. Almost no tutorial covers this. It should be table stakes.

A voice agent is four jobs, plus a conductor

Strip away the vendor branding and every voice agent does four things in a loop: it hears (speech-to-text), thinks (an LLM), speaks (text-to-speech), and connects (telephony). Audio comes in, becomes text, the model replies, the reply becomes audio, and it flows back out until someone hangs up.

The fifth piece is the conductor (Pipecat, in this build) that wires those four into one streaming loop with interruption handling and turn-taking. That orchestration layer is exactly what managed voice platforms sell. Hold onto that, because it is the whole cost story.

The hard part nobody warns you about: turn detection

Knowing when the caller has finished talking is the single hardest problem in the stack. Answer too early and you talk over people. Answer too late and the call feels dead. Kwindla Hultman Kramer, creator of the Pipecat framework, notes in his voice AI primer that the most common complaint about from-scratch agents is that they “interrupt too often.” There is academic literature on endpointing, which tells you this is not a solved-by-a-silence-timer problem. The scale of the pain is measurable: in AssemblyAI’s survey of 455 voice agent builders, 52.5% named accuracy and misunderstanding as their single biggest challenge, ahead of cost and latency.

The subtlety that took me longest to internalize: turn detection is actually two different jobs.

  • Barge-in: detecting that the caller has started talking, so the agent can shut up mid-sentence. This needs to be instant, so it should run locally.

  • End-of-turn: detecting that the caller has finished, so the agent can reply. This needs to be smart, because humans pause mid-thought constantly, so it should use a model that reads both the audio and the words.

Farid Fadaie, writing up lessons from three production voice builds, frames the entire architecture problem as one question: “who owns the next turn?” I think that is exactly right, and my answer is: a local detector owns barge-in, and the STT provider owns end-of-turn. Part 2 shows the two lines of code that implement that split.

Design decisions, slot by slot

Here is every pick, what it costs, and the popular alternatives with my honest read on each. The theme throughout: for a live phone call, latency beats intelligence, and each component lives in its own file so any of these picks is a one-line swap if you disagree. (This section gives the short version; the full decision tree with tables and pick-if rules lives in the companion decision guide.)

Hear: AssemblyAI Universal-3 Pro streaming ($0.45/hr, ~$0.0075/min)

Its neural turn detection is built into the model, which deletes the hardest problem in the stack from my to-do list, and its streaming models are promptable with up to 1,500 words of keyterms, so lead and product names come out right. The early-2026 Universal-3 Pro release moved to a speech-language-model architecture covering 99+ languages with code-switching, and streaming latency sits around 300ms on narrowband phone audio. The budget Universal-Streaming tier runs just $0.15/hr if cost matters more than the last point of accuracy. Docs here.

Alternatives worth your consideration: Deepgram is the other default for voice agents, and to be current about it, its Flux model (GA since October 2025, multilingual since April 2026) has model-integrated end-of-turn detection at roughly 260ms median, so built-in turn detection no longer separates the two. The matchup is a genuine coin flip decided by tiebreakers: keyterm prompting and unlimited concurrency favor AssemblyAI, while per-second billing and on-prem deployment favor Deepgram. Whisper is the open-source baseline with great accuracy, but it is batch-first, so you host it and fight for streaming latency yourself. Google Cloud STT has the widest language coverage and is convenient on GCP, but it is less tuned for low-latency turn-taking.

Think: Claude Haiku (a few cents of tokens per call; Haiku 4.5 at the time of writing, docs)

For a qualifying call, the caller never notices the difference between a good fast model and a great slow one, but they always notice the extra 500ms. Time to first token is the metric that matters, and a fast-tier model is plenty smart for a short structured conversation.

Alternatives: OpenAI GPT is the most common default across voice platforms, fast and strong at function calling, slightly pricier. Gemini Flash has a very fast first token, low cost, and a huge context window. Llama on Groq gives the lowest latency and cost of all when a small open model is enough. What I would avoid: reasoning-tier models. They pause to think, and on a live call those seconds read as a dead line.

Speak: Cartesia Sonic 3 (~$0.02 to $0.03/min effective)

Sonic 3 streams first audio in roughly 90ms, and the Turbo variant pushes that to about 40ms, the fastest published figures on the market (docs). That headroom pays for LLM think time, and the delivery survives a phone speaker. On a call, the voice starting instantly matters more than the voice being beautiful.

(One honest footnote: the repo pins CARTESIA_MODEL=sonic-2 as its tested default; trying Sonic 3 is a one-env-var change, which is rather the point of the architecture.)

Alternatives: ElevenLabs has the most expressive voices, its Flash v2.5 streams at ~75ms, and a May 2026 price reset cut its TTS rates by up to 55%, so the old “beautiful but expensive” framing is fading fast. Deepgram Aura-2 is built specifically for phone agents and offers on-prem for regulated work. Rime trains on real conversational speech for natural prosody, and it is literally a commented-out swap in my repo.

Connect: Twilio Media Streams (~$0.014/min outbound)

Not the cheapest, but the best-documented websocket media path (Media Streams docs) and the deepest ecosystem, which is what you want when a call is garbled at 11pm.

Alternatives: Telnyx runs a private backbone that shaves latency and cost, and is the standout alternative, just watch the per-transfer fees. Plivo publishes the lowest rates with an easy migration path. SignalWire offers Twilio-compatible APIs at lower cost.

The conductor: Pipecat (free, open source)

Pipecat wires the four services into one streaming pipeline that is, literally, a Python list. Interruption handling and turn-taking plumbing come built in, and every provider above is one line in the list.

Alternatives: LiveKit Agents is the other serious open-source framework, stronger on WebRTC-native use cases and large-scale infrastructure. For a telephony cascade like this one, Pipecat’s examples and integrations made it the faster path. Vapi and Retell are the managed options, and this is where the cost argument lives, so let us do the math.

Where the money actually goes

Add up my components: telephony ~$0.014 (plus a few percent in regulatory pass-through fees), STT ~$0.0075, TTS ~$0.03, LLM a cent or so. Call it $0.05 to $0.06 per minute, all-in, and closer to $0.04 if you pick the budget tier of each slot.

Now the managed route. Vapi advertises $0.05/min, but that is the orchestration fee alone. You still pay for the model, the voice, and the phone on top, and real deployments commonly land between $0.15 and $0.30 per minute. Retell at least publishes the honest math itself: a $0.055/min infrastructure fee inside an all-in range of $0.07 to $0.31 depending on the models you pick.

Read that again: the platforms pay the same component prices you do, then add roughly a nickel per minute for the orchestration layer. Pipecat does that layer’s job for free. You are not buying cheaper parts. You are skipping the tax.

Self-hosting also lets you cap the downside, not just the rate: a Twilio-enforced per-call time limit and a pipeline idle timeout (both in the repo, covered in Part 3) mean no stuck call can quietly run up minutes. Two costs the usual approach quietly adds on top: the no-code glue (n8n runs $20 to $60 a month on cloud plans, and it stops executing your workflows when you hit the plan’s execution cap, which is a spectacular failure mode for a lead-response system), and concurrency fees (Vapi includes about 10 simultaneous lines then bills ~$10 per line per month; Retell includes 20 then $8 per line, so an outbound campaign accrues fixed fees just for being busy).

The honest trade-offs

I want this series to be credible more than persuasive, so here is what the nickel does not buy you. You run a server, and the ops work of uptime, scaling, and the 2am page is real (Part 3 covers what that actually involves, and when paying the tax is rational). Managed platforms ship in an afternoon, and their turn-taking has been tuned across millions of calls; one production guide estimates roughly half of teams that start on managed platforms migrate to self-hosted within a year, which cuts both ways: managed is a fine place to start, and a place many teams outgrow. And at very low call volume the absolute dollar difference is small, so if you make fifty calls a month, pay the tax and ship.

But if you are calling leads at any real volume, and you want to actually understand the system answering your phone, build it. Part 2 walks through the code, following the path of a single call from form submission to transcript.

Next: Part 2, the implementation walkthrough →