Liz Zhang

Voice AI

The voice agent stack decision guide in 2026

By Liz Zhang · · 11 min read

← Back to blog

A companion reference to my build series (code, video). The series explains what I picked and why; this guide is the full decision tree, including the picks I did not make and exactly when you should make them instead. Prices and model names verified July 2026. This market moves quarterly, so treat the numbers as a snapshot and the decision rules as the durable part.

Every voice agent stack is one architecture question and five component decisions. Most comparison posts list features; this one gives you decision rules, because the honest truth about this market in 2026 is that the top options in each slot are close enough that your constraints, not the vendors’ benchmarks, should decide. Every pick below therefore carries its mirror image, a skip rule, because two options that are both “good if” is ambiguity, but one “good if” plus one “skip if” is a decision.

Three questions determine almost everything, so answer them first:

  1. Is the transcript load-bearing? If you need transcripts for compliance, QA, CRM write-back, or analytics (lead calling: yes, absolutely), you want a cascaded pipeline. If not, speech-to-speech models are in play.

  2. What is your latency budget? Under 800ms voice-to-voice is elite territory that constrains every pick. Under 1.5s is achievable with sane defaults. Over 2.5s means something is misconfigured.

  3. Volume and control. Under a few hundred calls a month with no in-house engineers, managed platforms are rational. Above that, or if you need to own the data path, build.

Decision 0: Cascade or speech-to-speech?

Before picking components, decide whether to have components at all. Native speech-to-speech models (OpenAI Realtime-2 as of May 2026, Gemini Live, Amazon Nova Sonic, xAI’s Grok Voice Agent) process audio to audio directly, with no transcript step, and they win on latency and emotional naturalness. Independent benchmarking teams like Coval put it plainly: S2S wins when the transcript is not load-bearing; cascades win on transcript availability, vendor swap-ability, and observability, and most production teams run both.

Pick a cascade if you need the transcript (compliance, QA, structured data capture), want to swap any component independently, or need to debug what the agent heard. Skip it if you are building a pure conversational experience, a companion, a game character, where feel is the whole product and no downstream system will ever read a transcript; there you are paying a pipeline tax for auditability you never use.

Pick S2S if you are building a conversational experience where feel beats auditability. For lead calling, where every call needs a consent trail and a CRM record, cascade is the clear answer, which is why the rest of this guide assumes one. Skip it if any system downstream needs structured data from the call (CRM, compliance, QA), because bolting a transcription pass onto S2S re-adds the cascade's cost without its control.

Decision 1: Orchestration, the fork that decides your cost structure

This is build vs buy, and it is worth more thought than the other four combined because it sets your per-minute economics and your ceiling of control.

Pick Pipecat if you write Python, your agent lives on phone lines, and you want the pipeline to be readable code. It has the deepest telephony-cascade example base, and its defaults are fast. Skip it if nobody on the team can own a Python service in production; the nickel you save per minute reappears as pager duty, and a managed platform failing is someone else's 2am.

Pick LiveKit Agents if your world is WebRTC-native (apps, browsers, multi-participant rooms) or you expect to scale to serious concurrent volume on their infrastructure. Skip it if your agent is a plain phone-line cascade; you would be taking on WebRTC infrastructure concepts for a problem Twilio websockets already solve.

Pick Vapi or Retell if you have no engineers to spare, need to ship this week, and your volume is low enough that a 3 to 5x per-minute premium costs less than the engineering time. Credit where due: Retell now publishes its honest all-in range ($0.07 to $0.31/min) itself, and includes 20 concurrent lines free with $8 per extra line, which is friendlier than it was a year ago. That is a legitimate trade at small scale. Know two things going in: the advertised fee excludes STT, LLM, TTS, and telephony, and roughly half of teams that start on managed platforms migrate off within a year, so structure your prompts and logic to be portable. Skip it if your volume is past a few thousand minutes a month or you need to own the data path; the 3 to 5x premium compounds monthly, and the migration off is the one half of teams end up doing anyway.

A third lane firmed up in 2026: single-vendor bundles. Deepgram’s Voice Agent API ($0.05 to $0.163/min by tier) and AssemblyAI’s Voice Agent API ($4.50/hr, one WebSocket for STT + LLM + TTS) collapse the whole cascade into one connection and one bill. They sit between managed platforms and hand-rolled pipelines: less markup than Vapi, less control than Pipecat. Worth a look if your team is one engineer and your traffic is real.

Decision 2: STT, the AssemblyAI vs Deepgram question

This is the matchup people actually search for, so let me give it the honest treatment: as of mid-2026 it is a genuine coin flip on quality and price, and the tiebreakers are specific.

Until recently the clean differentiator was turn detection: AssemblyAI’s Universal-3 Pro Streaming shipped with neural end-of-turn detection built into the model. That argument is now obsolete, because Deepgram’s Flux, GA since October 2025 with roughly 260ms median end-of-turn and multilingual (10 languages) since April 2026, solves the same problem natively. Both flagships now handle the hardest part themselves, and OpenAI even joined the streaming race in May 2026 with GPT-Realtime-Whisper at $0.017/min. Prices are nearly identical: AssemblyAI Universal-3 Pro Streaming at $0.45/hr ($0.0075/min), Deepgram streaming at $0.0077/min pay-as-you-go. On accuracy, independent benchmarks cluster the top tier within one to two points of each other, and the only honest answer to “which is more accurate” is: on your audio, measured, because results diverge on accents, noise, and domain terms far more than on leaderboards.

So decide on the tiebreakers:

Pick AssemblyAI if your calls are full of domain vocabulary. Its streaming models are promptable with up to 1,500 words of keyterms, which is the difference between “Metoprolol succinate” transcribed right and a cascading downstream error. The early-2026 Universal-3 Pro release also widened coverage to 99+ languages with code-switching, unlimited concurrent streams remain a real operational win, and the budget Universal-Streaming tier at $0.15/hr is the cheapest respectable streaming rate from a major vendor. Skip it if your calls are short and many (per-minute rounding quietly taxes 30-second calls) or the audio must stay on your infrastructure; it has no on-prem story.

Pick Deepgram if you want per-second billing (meaningful for short calls, where per-minute rounding quietly costs 30% or more), an on-prem or self-hosted deployment for regulated audio, the lowest-latency documented end-of-turn (~260ms median on Flux), or STT and TTS from one vendor (Aura-2 pairs with Flux on the same runtime, which trims a pipeline hop). Skip it if your transcripts hinge on niche vocabulary, drug names, SKUs, local street names, that you cannot afford to miss; keyterm prompting is the sharper tool for that, and it is not Deepgram's.

Pick Whisper (self-hosted or via Groq/Fireworks) if cost at very high volume or total control beats convenience, and you accept doing streaming engineering yourself; it remains batch-first. Skip it if you do not have infrastructure engineering time to spend; retrofitting streaming onto a batch-first model is a project, not a configuration.

The dark horse: ElevenLabs Scribe v2 Realtime posts ~150ms first-partial latency across dozens of languages and is worth benchmarking if multilingual real-time is your core constraint.

My build uses AssemblyAI, and the durable reasons are the keyterm prompting for lead and product names, the concurrency posture, and first-class Pipecat integration. If Deepgram fits your tiebreakers, nothing downstream in my architecture changes; it is a one-line swap, which is the point of the architecture.

Decision 3: LLM, where the metric is time-to-first-token

The counterintuitive rule: for short structured calls, pick the fastest acceptable model, not the smartest available one. Callers never notice the IQ difference between a good fast model and a great slow one; they always notice 500 extra milliseconds of silence.

Pick Claude Haiku or GPT’s fast tier as the default: quick first token, cheap, strong instruction-following, mature function calling for when the agent needs to book a slot or write a record. Skip the fast tier only if the call genuinely requires multi-step reasoning mid-conversation, and treat that as a design smell first: reasoning usually belongs before the call or after it, not on it.

Pick Gemini Flash if you want the lowest mainstream latency-to-cost ratio and a huge context window (useful if you stuff CRM history into the prompt). Skip it if you are not actually stuffing that context window; unused, the differentiator is gone and you have traded away your existing SDK and tool-calling patterns for nothing.

Pick Llama on Groq if you are chasing the absolute latency floor and a small open model covers your conversation complexity. Skip it if the conversation has edge cases a small model fumbles, objection handling, compliance-sensitive answers, because a wrong sentence costs you more than 100ms ever saved.

Avoid reasoning-tier models in the hot path entirely; their thinking pause reads as a dead line on a phone call. If you need heavy reasoning, do it before the call (research the lead) or after (analyze the transcript), not during.

Decision 4: TTS, latency vs beauty

Four serious contenders, and the trade is milliseconds against expressiveness.

Pick Cartesia if latency is your top constraint; Sonic 3 streams at ~90ms and Sonic Turbo at ~40ms, the market’s floor, and on a phone call that headroom pays for LLM think time. Worth knowing: Cartesia now also ships Ink (streaming STT at $0.13/hr, the cheapest respectable rate anywhere) and a full agents platform, so it is verticalizing fast. Skip it if the voice itself is the brand and expressiveness sells the product; you would be optimizing milliseconds the caller forgives while giving up the delivery they remember.

Pick ElevenLabs if voice quality is the product (its expressiveness leads the market); a May 2026 price reset cut TTS rates by up to 55% and Flash v2.5 streams at ~75ms, so the old cost and latency objections are much weaker than a year ago. Skip it if your calls ride narrowband phone lines and the voice is merely functional; the codec flattens the expressiveness you are paying for, and cheaper voices survive 8kHz just as well.

Pick Aura-2 if you want the cheapest phone-grade voice or a single-vendor stack with Deepgram. Skip it if the voice needs warmth beyond utility-grade or you are not otherwise in Deepgram's stack; alone, it is a bundle discount without the bundle.

Pick Rime if you want voices trained on real conversational speech; its Coda model streams sub-100ms, reads as pleasantly human on qualifying calls, and prices aggressively. Skip it if you need broad language coverage or a large stock-voice catalog; its range is narrower than the big two, and conversational prosody only wins in languages it ships.

One warning that applies to all of them: judge voices over an actual phone line, not your headphones. Narrowband 8kHz audio flattens differences you paid for and exposes artifacts you did not hear in the demo.

Decision 5: Telephony, the boring slot that bites

All four majors work. The differences are price, ecosystem, and edge-case fees.

Pick Twilio (~$0.014/min outbound, plus a few percent regulatory pass-through) as the default: the deepest docs, the largest example base, and the most battle-tested Media Streams path. When something breaks mid-call, the ecosystem is the product. Skip it if per-minute cost at volume is your deciding factor; the ecosystem premium is real money at a million minutes, and by then you have the in-house expertise it was compensating for.

Pick Telnyx if you are latency-obsessed or cost-optimizing at volume; its private backbone genuinely shaves both, just model the per-transfer fees if you do warm handoffs. Skip it if your call flows lean on warm transfers you have not priced (the per-transfer fees erode the savings) or your team debugs by searching community answers, which are scarcer here.

Pick Plivo for the lowest published rates on simple call flows. Skip it if your flows go beyond simple dial-and-stream; the lowest rate card comes with the thinnest media features and support, and you find out at the worst time.

Pick SignalWire if you want Twilio-compatible APIs at lower cost with more protocol control. Skip it if you do not actually need the extra protocol control; Twilio-compatible-but-smaller just means fewer answers when a call is garbled at 11pm.

One non-pricing factor that outweighs pricing for outbound: number reputation. Whichever carrier you pick, register your traffic, use branded caller ID where available, and monitor for “Spam Likely” flagging, because a flagged number silently destroys pickup rates and no per-minute saving compensates for calls nobody answers.

The cheat sheet

The meta-rule behind the whole table: architect so that every one of these is a one-line swap, then none of these decisions is expensive to get wrong. That is the real argument for the cascade, and it is worth more than any individual pick on this page.

Prices verified July 2026 against vendor pricing pages and independent trackers. If you are reading this later, verify before you model, and if a number here has drifted, tell me and I will update it.