Liz Zhang

Voice AI

The implementation: a lead-calling voice agent, following the path of one call (Part 2 of 3)

By Liz Zhang · · 10 min read

← Back to blog

Part 2 of a three-part series. Part 1 covered the case and the design decisions. This post walks the code. Part 3 covers latency, failure modes, and scaling. Full code: github.com/zhang-liz/OpenLine. Video walkthrough with timestamps: the YouTube series.

TL;DR: One Apps Script trigger, one FastAPI endpoint, and a Pipecat pipeline that is literally a Python list. The whole agent is a few hundred readable lines. This post walks it in the order a call happens, with the gotchas called out exactly where they bit me.

The best way to understand a voice agent is not file by file, it is call by call. So this walkthrough follows one lead from the moment they hit submit to the moment their transcript lands back in the sheet. Every snippet below is real code from the repo, trimmed only for clarity.

The stack, from Part 1: FastAPI server, Twilio Media Streams for the phone leg, Pipecat as the pipeline, AssemblyAI for speech-to-text and end-of-turn, Claude Haiku to think, Cartesia to speak, and a Google Sheet as the queue and the record.

1. The trigger: one guard, two problems

Everything starts inside the Google Sheet. An Apps Script onChange installable trigger fires the instant a new row is written, scans for rows that have a phone number but a blank status, and posts them to the server:

// apps_script.gs
for (let row = lastRow; row >= 2; row--) {
  const status = String(values[statusCol] || "").trim();
  const phone  = String(values[phoneCol]  || "").trim();

  if (phone && status === "") {
    // Mark "calling" immediately so a second onChange won't double-dial.
    sheet.getRange(row, statusCol + 1).setValue("calling");
    triggerCall(row);
  }
}

The status === “” condition is doing two jobs at once. A blank status means “new lead we have never called,” so it detects new work. And because the very first thing we do is stamp the row calling, and later the server writes back completed or failed, no row ever has a blank status twice. The system is idempotent by construction: no double dials, no infinite trigger loops, no separate state store. One condition, two problems solved.

This is also why the agent calls within seconds instead of minutes. The trigger is a push, not a poll. There is no timer checking the sheet every five minutes, which, as covered in Part 1, is exactly the lag that kills lead response. One subtlety the repo’s comments call out: it must be onChange, not the more familiar onEdit, because rows created by a form submission or an API are programmatic changes, and onEdit only fires on human edits. Pick the wrong trigger and your fastest leads, the ones arriving through the form, are exactly the ones that never trigger a call.

2. The gate: secret first, consent second

The Apps Script POSTs to a single endpoint. Two checks happen before anything dials:

# app.py
@app.post("/lead")
async def lead(request: Request, x_lead_secret: str = Header(default="")):
    if x_lead_secret != config.LEAD_SHARED_SECRET:
        raise HTTPException(status_code=401, detail="bad secret")

    row = (await request.json()).get("row")
    lead_row = sheet.get_lead(row)

    if not sheet.consent_given(lead_row):
        sheet.write_result(row, "failed", "skipped: no consent")
        return JSONResponse({"status": "skipped", "reason": "no consent"})

The shared secret matters more than it looks. This endpoint places outbound phone calls, which cost money and carry legal weight. Without the header check, anyone who discovers the URL has a free robocall machine billed to your Twilio account. Toll fraud against unguarded voice endpoints is common enough that this three-line check is non-negotiable.

The consent gate is the compliance layer from Part 1 made concrete: the agent structurally cannot dial a lead who did not opt in. Not “should not.” Cannot. If the flag is missing, the row is marked failed with a reason and the function returns before Twilio is ever touched.

One more small decision hiding here: the server passes the lead’s name along with the call as a stream parameter, so the agent can greet them without a second round-trip to the sheet mid-call. (Gotcha: Twilio strips query strings from Media Stream URLs, so custom parameters on the stream are the only reliable way to carry data onto the call. This cost me an evening. The downstream consequence is worth knowing: the websocket handler has to accept the connection before it knows which lead it is for, because Twilio sends two JSON frames first, connected and then start, and the row and name only arrive inside start.customParameters. If your handler assumes it knows the caller at accept time, it does not.)

3. The pipeline: the whole agent is a list

Twilio dials, the lead answers, and Twilio opens a websocket to the server carrying raw call audio. That socket gets handed to Pipecat, and this is the part I most want you to see, because this list is the voice agent:

# pipeline.py
pipeline = Pipeline([
    transport.input(),        # caller audio in from Twilio
    stt,                      # speech -> text (AssemblyAI)
    transcript.user(),        # log what they said
    aggregator.user(),        # add it to conversation memory
    llm,                      # think (Claude Haiku)
    tts,                      # text -> speech (Cartesia)
    transport.output(),       # agent audio back to the caller
    transcript.assistant(),   # log what we said
    aggregator.assistant(),   # remember it
])

Audio flows top to bottom, in a loop, until someone hangs up. When people ask what a managed voice platform actually does for its per-minute fee, the honest answer is: mostly this list. Reading it is also the fastest way to understand any voice agent architecture, including the ones you did not build.

Every service in the list comes from a tiny factory in its own file (make_stt(), make_llm(), make_tts()), which is what makes providers one-line swaps. The repo ships with a commented-out Rime block at the bottom of tts.py: uncomment it, comment the Cartesia one, restart, different voice. That is the no-lock-in claim from Part 1, demonstrated in ten seconds.

For senior engineers: frames and back-pressure

If you plan to extend the pipeline, the mental model that matters is that Pipecat moves typed frames, audio chunks, text fragments, and control signals, through the processor list in order, and interruptions are implemented as control frames that flush everything queued downstream (architecture docs). Two consequences fall out of that. First, processor order is semantic: the context aggregator must sit where it does so a barge-in cancels the half-spoken response before it is committed to memory, or the model will believe it said things the caller never heard. Second, streaming works because no stage waits for a complete payload, which means any custom processor you insert must handle partial frames and pass through frames it does not own, or you will silently break interruption for everything after it.

4. Turn-taking: two jobs, two tools

Part 1 argued that “when do I speak” is really two different problems. Here is the entire implementation of that split.

Barge-in, letting the caller cut the agent off, is handled by a small local voice activity detector (Silero VAD) on the transport:

# pipeline.py -- inside the transport params
vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),

The moment the caller starts talking, this fires locally, with no API round-trip, and interrupts the agent’s speech mid-sentence. That immediacy is why it runs on your own server.

End-of-turn, knowing the caller is actually finished, is delegated to AssemblyAI’s neural endpointing, configured in the STT service:

# stt.py
AssemblyAIConnectionParams(
    speech_model="u3-rt-pro",
    end_of_turn_confidence_threshold=0.7,
    min_end_of_turn_silence_when_confident=160,
    max_turn_silence=700,
)

The model reads both the audio and the words to decide when a turn is over, so the agent waits through a mid-thought pause but replies quickly after a real one. These three thresholds are the dial you will actually tune in testing: tighter feels snappier but risks clipping people, looser never interrupts but drags. Do not add a second VAD in front of this for end-of-turn; the local detector is only there for barge-in.

5. The first word: skip the model entirely

The single highest-leverage latency trick in the build is that the agent’s opening line never touches the LLM:

# pipeline.py
@transport.event_handler("on_client_connected")
async def _on_connected(_transport, _client):
    await task.queue_frames([TTSSpeakFrame(greeting)])

The greeting is a fixed string, an intro plus a permission ask (“Is now a good time?”), spoken directly through TTS the instant the call connects. The lead hears a warm, human-sounding hello with zero model latency, and the LLM only runs once they reply. The greeting is also seeded into the conversation context as an assistant turn, so the model knows it has already introduced itself and does not do it twice.

Small trick, outsized effect: the first two seconds of an outbound call decide whether the person stays on the line.

An enhancement worth making right here, before you point this at real leads: use the greeting to disclose. The repo’s opener is a plain intro-plus-permission ask, but the greeting is also the natural place to tell the lead two things a growing number of jurisdictions now expect them to hear, that the call is recorded and that they are speaking with an AI. Because the opener is one templated constant spoken straight through TTS, adding disclosure is a one-line edit to it:

# prompt.py -- the GREETING constant, disclosure added inline
GREETING = (
    "Hi {name}, this is Chloe, an AI assistant from Bayhaus Creative calling on "
    "a recorded line about the form you just submitted. Is now a good time?"
)

Treat this as product design, not legal fine print, and weigh two things. Disclosing up front reads as honesty, and honesty tends to help the call: the leads who would have bailed the second they realized it was a bot self-select out in the first two seconds instead of burning a minute, and the ones who stay are talking to you on clear terms. The mechanics are already handled, pipeline.py seeds build_greeting() into the context as the assistant’s first turn, so editing this one constant keeps what the caller hears and what the model thinks it already said in lockstep, with no second change. What you must disclose, recording versus AI, and in what words, varies by region and is not legal advice from me, so if you call across jurisdictions, lift the line into config and key it to the lead’s location rather than hard-coding one sentence for everyone.

While we are near the prompt: prompt.py treats the system prompt as a latency tool, not just a persona. Its hard rules cap every turn at one short sentence, ban one question stacking on another, and blacklist filler acknowledgments outright (“Great!”, “Absolutely”, “That makes sense”), because every word the model does not generate is a word TTS does not have to speak. On a phone call, prompt discipline is a performance optimization wearing a UX costume. It also carries no per-lead data, the name rides in the greeting, so the whole prompt stays a stable prefix, which is exactly what lets Part 3 cache it and stop paying for that reference block on every turn.

6. The cleanup: hang-ups must be handled, not hoped away

When the caller hangs up, Twilio closes the websocket. The pipeline has to be explicitly cancelled when that happens, so the run loop exits and the transcript writes back to the sheet. Skip this and the pipeline sits there, services politely trying to reconnect, and the write-back never fires. Your sheet fills with rows stuck on calling and you spend an hour learning what I just told you in a sentence.

That write-back closes the loop from Part 1, and it is actually two paths, deliberately split. The websocket handler writes completed plus the full transcript, but only when the call produced real dialogue. A separate Twilio status callback catches everything else, mapping no-answer and busy to no-answer and failed and canceled to failed, so a lead who never picked up can never end up looking like a completed conversation in your data. Status and transcript land in the same row the lead arrived on: your call record, your QA data, and your compliance trail in one place.

What you have now

A form submission that becomes a live, natural phone call in seconds, with consent enforced, interruptions handled, and a transcript filed, in a few hundred lines of Python you can read in one sitting. Clone the repo, drop in your keys, point it at a sheet, and call yourself first.

It works. Which is exactly when the interesting problems start. Part 3 is the production post: the latency budget and where the milliseconds actually go, the four ways voice agents break in the wild, and why the obvious way to scale this is wrong.

Next: Part 3, latency, failure modes, and scaling →