Building Voice Agents

In L252 (Audio: Speech-to-Text & Text-to-Speech) you wired up the cascaded voice loop — microphone → STT → LLM → TTS → speaker — and learned to measure transcripts with WER, guard against silence hallucinations with VAD, and stream every stage to land under the latency budget. That gives you a system that can hear and speak. It does not give you a system you can have a conversation with.

The difference is everything. A demo that transcribes your sentence, thinks, and reads a reply is a walkie-talkie: you talk, you wait, it talks. A real voice agent is full-duplex and interruptible — it knows the instant you stop talking (so it doesn't leave dead air or cut you off), it lets you barge in and stops mid-sentence the moment you do, and it recovers gracefully when the conversation doesn't go to script. As the saying goes among voice engineers: the LLM in the middle is the easy part — the hard part is the turn-taking.

This lesson is about that conversation-control layer: the logic that sits around the L252 loop and makes it feel like talking to a person instead of a phone tree.

In this lesson:

• Turn-taking & endpointing — how the agent decides your turn is over (VAD vs. semantic turn models) and why a fixed silence timeout is a trap.
• Barge-in — the defining feature: stop the agent fast and truncate its transcript to what you actually heard, or its memory desyncs from reality.
• The voice-to-voice latency budget — where the ~600ms actually goes, and why it hides in turn-taking and the LLM, not STT/TTS.
• Dialogue state & tools in voice — function-calling mid-conversation without dead air.
• Who owns the loop — cascaded (you do, with a Claude brain) vs. realtime speech-to-speech (the provider does), and the 2026 stack (LiveKit, Pipecat, Vapi; OpenAI Realtime, Gemini Live).
• Evaluating voice agents — why you have to simulate conversations, not unit-test them.

Scope: this lesson builds directly on L252 (audio STT/TTS, cascaded vs. speech-to-speech, the latency budget), L224 (latency as a feature), L225 (streaming UX), and L249 (observability). It defers image & video generation to L254 (Image & Video Generation) and multimodal retrieval to L255 (Multimodal RAG & Pipelines). STT/TTS internals (WER, diarization, SSML) stay in L252 — here we treat them as building blocks and focus on the live loop.

Picture the L252 loop running as a turn-based system. You speak; the agent records until it thinks you're done; it transcribes, calls the LLM, synthesizes speech, and plays it; then it listens again. Each box works. Stitched together naively, the conversation still feels broken — and always for the same three reasons.

1. It doesn't know when you're done. Humans hand off the floor with gaps of only 200–300ms. A naive agent waits for a fixed window of silence — and most production agents land at 800–1500ms of perceived gap. Anything past ~1 second and callers either repeat themselves (now you have two overlapping utterances) or assume the line dropped and hang up. Set the window shorter to fix the lag and you create the opposite failure: it cuts people off mid-sentence the moment they pause to think.

2. You can't interrupt it. When a human is talking and you say "no, wait —", they stop. A voice agent that finishes reading its scripted paragraph while you're trying to correct it is infuriating, and it's the single fastest way to make a bot feel like a bot. Real conversation requires barge-in.

3. When it breaks, it breaks badly. Interruptions, tool calls that take a second, background noise, a user who changes their mind — the agent has to keep the dialogue state straight through all of it, and keep its own memory aligned with what the user actually heard.

So a voice agent is best understood as two layers:

• the media layer you already built (STT, the LLM, TTS — what is said), and
• the conversation-control layer (turn-taking, barge-in, dialogue state — when and who holds the floor).

L252 was the media layer. This lesson is the control layer. A useful framing to keep throughout: turn detection decides when speech started or ended; interruption handling decides what the agent does with that signal. They are different jobs, and conflating them is where most voice agents go wrong.

Endpointing is the decision: the user has finished their turn; start thinking. Get it wrong in either direction and the conversation falls apart. There are three layers of sophistication, and modern agents stack them.

Layer 1 — VAD (Voice Activity Detection). VAD works at the audio level: it classifies each incoming audio frame as speech or non-speech, continuously, even before any transcription happens. Silero VAD is the de-facto standard. VAD answers "is someone making speech sounds right now?" — it does not know meaning. On its own, VAD-based endpointing is just a timer: speech stopped → wait N ms of silence → declare the turn over.

Layer 2 — the silence-timeout trap. That timer N is a single number that you cannot win with. The tradeoff is brutal and unavoidable: a silence timeout of 800ms adds nearly a full second to every single response before the pipeline even starts. Make it shorter to feel snappy and you get premature endpointing — the user says "I'd like to book a table for two, at...", pauses half a second to recall the time, and the agent barges in over them because the timeout was 250ms. Make it longer to stop cutting people off and now every turn carries that dead-air tax. One fixed number serves a fast "yes" and a slow "my account number is... four... seven..." equally badly.

Layer 3 — semantic (model-based) turn detection. The 2026 answer escapes the tradeoff by adding meaning. A small, fast turn-detection model reads the partial transcript in real time and predicts whether it looks like a finished thought. "I'd like to book a table for two, at" is obviously incomplete — hold the floor, even through a long pause. "...at seven pm." is obviously complete — release immediately, before the silence timer would even fire. Production examples: LiveKit's TurnDetector (a ~135M-parameter SmolLM-v2 fine-tune), Pipecat's Smart Turn v3 (a local turn analyzer), and Deepgram Flux, which bakes turn detection into the speech-recognition model itself so the same model uses acoustic and semantic context (configurable via eot_threshold, eager_eot_threshold, and eot_timeout_ms).

The production architecture is two signals working together: acoustic VAD (fast, ~no latency, catches that sound started/stopped) plus a semantic turn model (catches whether the thought is finished). Semantic detection alone over-triggers on genuinely incomplete sentences and over-reacts to mid-sentence pauses; VAD alone is the silence-timeout trap. Together they give natural pacing.

Backchannels are not turns. When you say "uh-huh", "yeah", "right" while the agent talks, you are not taking the floor — you're signalling "I'm listening, keep going." A bare VAD hears speech energy and (wrongly) treats it as an interruption; a good turn model classifies the backchannel and the agent keeps talking. Distinguishing backchannel vs. barge-in vs. noise is exactly the learned signal that dedicated turn-detection models provide.

Barge-in — letting the user cut the agent off mid-sentence — is the single feature that separates a voice agent from a text-to-speech announcement. It sounds simple ("stop talking when the user talks") and it is the part teams most often get subtly, conversation-breakingly wrong. Done right, it's three steps, not one:

1. Stop playback — fast. The moment turn detection fires during agent output, cancel the current TTS stream and hand the floor back to the input pipeline. The target is roughly 25ms to clear the pipeline — any slower and the agent talks over the user for a beat, which feels worse than no barge-in at all. (This is also why you need acoustic echo cancellation on the client: without it the agent hears itself through the speaker and barges in on its own voice.)

2. Truncate the transcript to the heard boundary — this is the step everyone forgets. Your TTS was ahead of the audio: when you stopped the speaker, the model had already "said" (in its conversation history) a sentence the user never heard. You must trim the assistant message in context to only the audio that was actually played. If you skip this, the model's memory desyncs from reality: it thinks it offered "we have a 7pm and a 7:30", the user only heard "sure, I can book that", and the next turn the agent says "like I said, 7 or 7:30" about times it never spoke. "Playback truncation not reflected in conversation history" is the classic post-interruption failure — it produces an agent that gaslights its own callers.

3. Commit the new user turn — with task state preserved. The interruption opens a fresh user turn. Re-run endpointing on it, but keep the task state (the booking in progress, the slots already filled) so a correction ("no, Friday") edits the task instead of restarting it.

Not every sound is a barge-in. A robust agent classifies the input before reacting: a correction ("no, Friday") → stop and accept the turn; a backchannel ("yeah", "uh-huh") → keep talking; noise (a keyboard click) → resume playback and log a false positive; a DTMF keypress → route it; a silence timeout → re-prompt. The biggest real-world bug is false barge-in — noise, echo, or an over-sensitive VAD makes the agent stop repeatedly, and the conversation stutters to a halt.

On a native speech-to-speech provider, much of this is handled server-side. The OpenAI Realtime API exposes the exact lifecycle as events — the model emits input_audio_buffer.speech_started when it detects you talking, you cancel the in-flight response, and you truncate the assistant item to the audio actually played. The mechanic is identical whether you implement it or the provider does:

A conversation feels natural when the reply lands inside roughly a 1-second voice-to-voice window (measured from you stop talking to you hear the first sound back); humans themselves hand off in 200–300ms. Under ~1.5s feels good, 1.5–2.5s is typical production, and past ~3s it feels broken. Here's where one turn actually goes in 2026:

A representative cascaded turn: 50 (network) + 200 (turn-taking) + 80 (STT) + 250 (LLM TTFT) + 80 (TTS TTFA) ≈ 660ms to first sound.

The non-obvious lesson: STT and TTS are not where your latency hides. The two places the time actually goes are turn-taking and LLM time-to-first-token. That reframes optimization: tightening endpointing (a semantic turn model that fires the instant you finish, instead of an 800ms timer) and shrinking TTFT (a faster model, prompt caching, a shorter system prompt) buy you far more than swapping STT vendors.

And it all rests on streaming every stage (the L252 lesson, applied live): STT emits partial transcripts roughly every ~50ms instead of waiting for a full sentence; the LLM streams tokens as it generates; TTS streams audio chunks before the full reply text exists. The agent starts speaking the first sentence while the LLM is still writing the second. Turn streaming off anywhere and the budget blows past seconds — batch is for offline transcription, never for live talk.

A voice agent that can only chat is a toy; a useful one does things — checks availability, books the table, looks up an order. That means function-calling (tools) from L-series tools lessons, but with two voice-specific twists.

1. Tools take time, and silence is unacceptable. In text, a 1.5s tool call is invisible. In voice, 1.5s of dead air sounds like the line dropped. The pattern is to speak a filler while the tool runs — "let me check that for you..." — so there's continuous audio. The best speech-to-speech models can call functions in the background while the model keeps talking; in a cascaded agent you emit the filler utterance, fire the tool, and stitch the result into the next turn.

2. State must survive interruptions. The dialogue/task state (which slots are filled, what's confirmed) is separate from the raw transcript. When the user barges in to correct a detail, you edit the task state — you don't restart the booking. This is why barge-in step 3 preserves task state.

Because Claude has no native audio (L252), in a voice agent Claude is the brain in a cascaded pipeline: a framework (LiveKit, Pipecat) owns the mic/VAD/turn-taking/STT/TTS, and calls Claude as the reasoning LLM. Claude Sonnet 4.6 is a strong fit — fast, and reliable at multi-turn instruction-following and policy adherence over long conversations, which is exactly what a live agent stresses. Here's a cascaded voice agent wired with LiveKit, Claude as the brain, and a tool:

L252 framed the architecture choice as cascaded (STT→LLM→TTS) vs. speech-to-speech in terms of control, observability, and cost. For a live agent, the same choice reduces to one sharp question: who owns the turn-taking and barge-in loop — you, or the provider?

Cascaded — you own the loop. A framework + your choice of STT, LLM, and TTS. You implement (or configure) endpointing, barge-in, and truncation. You get any LLM brain (Claude Sonnet 4.6), per-stage traces (L249 — you can see the WER, the TTFT, the exact truncation point of every turn), provider flexibility, and predictable cost. The price: you own the hard control logic, and you stitch more moving parts.

Realtime / speech-to-speech — the provider owns the loop. A single model takes audio in and emits audio out, handling VAD, turn detection, and interruption server-side. OpenAI's Realtime API (the gpt-realtime family, now out of beta and built for production voice agents, with GPT-5-class reasoning in the latest tier) and Google's Gemini Live API (sub-400ms first audio, multimodal voice+vision) are the leaders. You get the lowest latency (~250–500ms first audio) and natural prosody for free — at the cost of vendor lock-in, no transcript / per-stage traces, and higher price.

The framework layer (cascaded):

• Vapi — a managed platform: configure STT/LLM/TTS in a dashboard and it runs the listen→think→speak loop. Fastest path to a phone agent; best when you'd rather configure than code.
• Pipecat — open-source Python (by Daily, v1.0 in 2026), modelled as a pipeline of frame processors (audio frames flow through VAD→STT→LLM→TTS). Maximum control over every step; PipelineParams(allow_interruptions=True) plus Smart Turn v3 + Silero VAD is the default barge-in setup.
• LiveKit Agents — open-source WebRTC infrastructure + an Agents SDK, with a built-in TurnDetector model and SIP telephony. Best for multi-participant rooms and self-hosting.

Cost crossover: above roughly 10,000 minutes/month, running the framework path (LiveKit/Pipecat) tends to undercut managed platforms by 60–80% per call; below that, managed is usually cheaper once you price in engineering time. Prototype on managed; graduate to a framework when volume justifies owning the stack.

You cannot unit-test a voice agent the way you test a function, because the user is part of the loop — pacing, interruptions, accents, noise, and emotion all change the outcome. The 2026 discipline is conversation simulation: a test harness role-plays callers (with accents, background noise, and deliberate interruptions) against your agent, at scale, and scores the whole interaction.

What to measure (across the stack):

• Transcription quality — WER on your audio (the L252 metric), because a wrong transcript poisons everything downstream.
• End-of-turn accuracy — how often endpointing fired correctly vs. cut the user off or lagged.
• Interruption handling — did barge-in stop fast, and did the truncated context stay consistent? Post-interruption recovery is its own axis (benchmarks like IHBench test whether the agent keeps making progress through a task after being interrupted).
• Conversational pacing / latency — P50 and P95 voice-to-voice latency.
• End-to-end task outcome — did the booking actually get made, correctly?

The trap that catches everyone: a latency dashboard says "P95 turn latency is healthy" while callers are hearing the agent cut them off — because aggregate latency says nothing about interruption quality. The release gate is whether QA can replay the specific interrupted turn with its audio, transcript, and trace pointers (L249 observability, applied to voice). Tools built for this: Hamming (scenario generation + speech-level analysis of frustration, pauses, and interruptions; 1000+ concurrent simulated calls), Coval (CI/CD, autonomous-vehicle-style simulation), and Cekura (automated test generation). And the flywheel from L246 (CI/CD for prompts & evals) applies verbatim: every production failure becomes a new simulated test case.

Reading about endpointing and barge-in is one thing; feeling the tradeoffs is another. The lab below lets you drive the two decisions that make or break a voice agent.

Start with ① YOUR TURN: the default is a 250ms silence timeout against a 600ms mid-sentence pause — watch it cut you off. Now raise the timeout until it stops cutting you off, and notice the dead air it adds to every turn. Then switch to Semantic turn (Smart Turn) and see it hold the floor through your pause and fire fast — the tradeoff disappears.

Then ② AGENT SPEAKS: pick Barge in and click any word of the agent's reply to interrupt there. With interrupt handling ON, the transcript truncates to what you actually heard; toggle it OFF to watch the model's memory desync from reality. Try a Backchannel "mm-hm" under each endpointing mode — semantic keeps talking, a bare VAD false-stops. Watch the voice-to-voice latency bar the whole time, and flip Cascaded ↔ Realtime to see who owns the loop.

Notice three things. One: a fixed silence timeout can never be right — it's always too short for a slow speaker or too long for a fast one; only the semantic turn model escapes the tradeoff. Two: barge-in isn't just "stop talking" — without the transcript truncation, the agent's next turn references things you never heard. Three: the latency lives in turn-wait and LLM TTFT — the amber and violet segments — not in STT or TTS.

Predict first, then check in the lab (or reason it through).

1. Your agent uses a 300ms silence timeout. A caller reading a credit-card number pauses ~700ms between digit groups. What happens, and what are your two fixes?

2. A user barges in after hearing "Sure, I can book that —" but your code cancels TTS without truncating the transcript. The model's history still contains "...we have a 7pm and a 7:30. Which works for you?" What does the agent say next, and why is it a disaster?

3. Your voice-to-voice P95 is a healthy 850ms, but users complain the bot "talks over them." Your latency dashboard is green. Where do you look?

4. You need a phone agent live by Friday for a low-volume internal tool, and your team is two people. Cascaded framework or managed platform — and why?

Answers.

1. The 300ms timeout is shorter than the 700ms pause, so the agent endpoints prematurely and interrupts the caller mid-number — then transcribes a fragment. Fixes: (a) raise the silence timeout for this flow (accepting more dead air — patient endpointing suits long account/card numbers), or far better (b) use a semantic turn model that knows "4 7 1 2 ..." is an unfinished sequence and waits for the whole number. (Bonus: some agents switch endpointing profiles by dialog state — patient while collecting a number, snappy during chit-chat.)

2. Next turn the agent says something like "As I mentioned, 7 or 7:30 — which do you prefer?" — referring to options the user never heard. The model's memory desynced from the played audio because you skipped transcript truncation. The user is confused or feels gaslit. Always truncate the assistant message to the heard-audio boundary on every barge-in.

3. Aggregate latency hides interruption quality. "Talks over them" is a barge-in / endpointing problem, not a latency one — the agent isn't stopping fast enough (or an over-sensitive/under-sensitive VAD is misfiring). Pull the specific interrupted turns and replay them with audio + transcript + trace (L249); measure barge-in stop time and end-of-turn accuracy, not just P95.

4. Managed (e.g. Vapi). At low volume with a tiny team and a Friday deadline, the managed platform's handled telephony + turn-taking saves far more than it costs; the framework path only pays off past ~10k min/month where it undercuts managed by 60–80%. Prototype managed, graduate later.

• "A voice agent is just STT → my chatbot → TTS." → That's a walkie-talkie. A real agent adds the conversation-control layer — turn-taking, barge-in, recovery. The LLM is the easy part.
• "Endpointing is just waiting for silence." → A fixed silence timeout is a trap: too short cuts people off, too long adds dead air to every turn. Semantic turn detection (read the unfinished thought) escapes the tradeoff; pair it with VAD.
• "Barge-in means stop the audio." → Stopping is step one of three. You must also truncate the transcript to the heard audio (or the model's memory desyncs) and commit the new turn with task state preserved.
• "Any speech while the agent talks is an interruption." → No — "uh-huh" is a backchannel (keep talking), not a barge-in. Treating backchannels as turns makes the agent stop constantly.
• "Most of the latency is in STT/TTS." → It's in turn-taking and LLM time-to-first-token. Optimize endpointing and TTFT, not your STT vendor.
• "Speech-to-speech is strictly better — it's faster." → It's the lowest latency and the provider owns the loop, but you give up the transcript, per-stage traces, provider choice, and pay more. Cascaded still wins for control, observability, and compliance.
• "If P95 latency is green, the agent feels good." → Aggregate latency says nothing about whether it cuts people off. Evaluate interruption handling and end-of-turn accuracy by replaying real interrupted turns.
• "I can unit-test my voice agent." → The user is in the loop. You must simulate conversations (accents, noise, interruptions) at scale and score the whole interaction.

• A voice agent is the L252 cascaded loop made live and interruptible. The media layer (STT/LLM/TTS) is the easy part; the conversation-control layer — turn-taking, barge-in, recovery — is the hard part and the whole job.
• Turn detection decides when a turn ends; interruption handling decides what to do about it. Keep them distinct.
• Endpointing: a fixed silence timeout forces a lose-lose between cutting users off and dead air. The 2026 answer is two signals — acoustic VAD (Silero) + a semantic turn model (LiveKit TurnDetector, Pipecat Smart Turn, Deepgram Flux) — that reads the unfinished thought.
• Barge-in is three steps: stop TTS in ~25ms, truncate the transcript to the heard audio (or the model's memory desyncs from reality), and commit the new turn with task state preserved. Backchannels ("uh-huh") are not barge-ins. Echo cancellation stops the agent interrupting itself.
• Latency budget (~600ms–1s voice-to-voice): the time hides in turn-taking and LLM TTFT, not STT/TTS. Stream every stage; speak a filler while tools run.
• Who owns the loop: cascaded (LiveKit/Pipecat + Deepgram + Claude Sonnet 4.6 + Cartesia) gives you a tunable brain, per-stage traces, and provider choice; realtime speech-to-speech (OpenAI Realtime, Gemini Live) gives lowest latency and provider-owned turn-taking, at the cost of lock-in and observability. Prototype on Vapi; graduate to a framework past ~10k min/month.
• Evaluate by simulation — accents, noise, and interruptions at scale (Hamming, Coval, Cekura) — and score end-of-turn accuracy, interruption recovery, pacing, and task outcome, not just aggregate latency. Feed every production failure back as a test case (L246).
• Next — L254 (Image & Video Generation: Diffusion Basics): we leave the conversational stack for the generative side of multimodal — how diffusion models turn text into images and video, and how to wire them into a product.