Skip to main content
CodePeetCodePeet
Production AI Engineering (LLMOps)

Audio: Speech-to-Text & Text-to-Speech

Introduction

You've given your AI eyes (L250) and taught it to read documents (L251). Now you give it ears and a voice. Audio is the modality behind the products people increasingly expect: voice assistants, meeting transcribers, call-center copilots, accessibility tools, dictation, and — the big one — conversational voice agents that talk back in real time.

Audio comes down to two pillars, and this lesson is about each one and how to chain them:

  • Speech-to-Text (STT), a.k.a. ASR (Automatic Speech Recognition): audio in, text out.
  • Text-to-Speech (TTS): text in, audio out.

Put an LLM between them and you have the classic voice loop: 🎤 → STT → LLM → TTS → 🔊. Most of the engineering difficulty isn't the LLM in the middle — you already know that part. It's the two audio ends (how accurate is the transcript? how natural and fast is the voice?) and the brutal latency budget of holding a real conversation. A note up front: Claude is a text + vision + document model — it doesn't do native audio I/O, so for audio you pair it with specialized providers (OpenAI, Deepgram, AssemblyAI, ElevenLabs, Cartesia, Gemini). The concepts below are provider-agnostic.

In this lesson:

  • STT: how ASR works, the WER metric, streaming, diarization, timestamps, and VAD
  • STT failure modes — noise, accents, homophones, and hallucination on silence
  • TTS: voices, expressiveness, SSML, voice cloning, and time-to-first-audio
  • The latency game — the ~300ms conversational budget and why you stream every stage
  • The architecture choice — cascaded (STT→LLM→TTS) vs a native speech-to-speech model

Scope: this lesson is the building blocks — STT and TTS, and the pipeline that chains them. It defers the full conversational agent — turn-taking, barge-in (interrupting), endpointing strategy, and dialogue management — to L253: Building Voice Agents, which assembles these blocks into a live agent. It also defers image/video generation to L254 and multimodal RAG to L255, and builds on latency-as-a-feature (L224) and streaming UX (L225).

Two-panel hero infographic titled 'Audio: Speech-to-Text & Text-to-Speech'. The LEFT panel, 'TWO PILLARS, ONE PIPELINE', shows the cascaded voice loop with real arrows: a microphone feeds speech-to-text (STT/ASR), whose transcript feeds your LLM, whose reply feeds text-to-speech (TTS), which feeds a speaker. It states the quality and latency facts for each pillar. STT: measured by Word Error Rate (WER = substitutions + deletions + insertions over reference words), with state-of-the-art around 5 to 6 percent on clean audio; you also get word timestamps and speaker diarization, and you must run Voice Activity Detection to strip silence or the model hallucinates phantom phrases on non-speech. TTS: judged by voice quality, expressiveness, and time-to-first-audio (the fastest models reach roughly 40 to 100 milliseconds), with SSML or audio tags needed so numbers and acronyms are pronounced correctly. The RIGHT panel, 'THE LATENCY GAME — AND THE BIG CHOICE', makes the point that humans expect a reply within a roughly 300-millisecond budget, so you must STREAM every stage — STT emits a near-final transcript, the LLM emits its first token, and TTS starts speaking before the sentence is finished — to land a well-built streaming pipeline around 400 to 700 milliseconds to first sound; batch processing takes seconds and is only for offline transcription. It then contrasts the two architectures: a CASCADED STT-LLM-TTS pipeline wins on control, observability (per-stage traces), provider choice, and predictable cost, while a native SPEECH-TO-SPEECH model is the lowest latency (around 250 to 350 milliseconds) but gives up the transcript and per-stage debugging, locks you to one vendor, and can be far more expensive. The footer reads: STT in, TTS out, stream every stage to hit the conversational budget, and choose cascaded for control or speech-to-speech for latency.

Two Pillars, One Pipeline

Before the details, hold the whole loop in your head, because every decision trades against the others:

🎤 audio  →  [STT]  →  text  →  [LLM]  →  text  →  [TTS]  →  🔊 audio

Each arrow is latency and each box is cost and a quality decision. The transcript quality (STT) determines what the LLM even gets to reason about — garbage in, garbage out. The voice quality and speed (TTS) determine whether the result feels like a person or a robot reading. And the sum of the stage latencies determines whether a conversation feels natural or whether the user keeps talking over an awkward silence.

This cascaded design — three separate services you wire together — is the dominant production architecture, and for good reason (control, observability, provider choice — more on that later). But it's not the only option: a newer class of speech-to-speech models collapses all three boxes into one. We'll build up the cascaded version first (because understanding the pieces is the point), then compare. Start by mastering each pillar.

Speech-to-Text — How It Works, and How to Measure It (WER)

Modern ASR is a neural model (Whisper-style transformers, and provider models like Deepgram Nova, AssemblyAI Universal, ElevenLabs Scribe) trained on enormous amounts of transcribed audio. You give it an audio stream; it gives you text — and, importantly, a lot more than just text: word-level timestamps, confidence, language detection, and (with the right model) speaker diarization (who said what).

The metric you live by is Word Error Rate (WER). It's the edit distance between the transcript and the ground truth, normalized by length:

WER = (Substitutions + Deletions + Insertions) / N — the number of words wrong, divided by the number of words actually spoken.

Lower is better; 0% is perfect. On clean, well-recorded English, today's best models are around 5–6% WER — roughly one word in twenty. That sounds great until you remember two things: (1) WER is brutal on the worst audio, climbing past 20–30% on noisy or heavily-accented speech, and (2) the kind of error matters — a deleted ‘um’ is harmless; a substituted ‘1,500’→‘1,500’ → ‘15,000’ is a disaster. Always evaluate WER on your audio (your accents, your jargon, your microphones), not the vendor's clean benchmark — and weight the words that matter (names, numbers, key terms).

STT in Practice — Streaming, Diarization, Timestamps, and VAD

Three practical decisions define an STT integration:

  • Streaming vs. batch. Batch sends a whole audio file and waits for the full transcript — perfect for offline work (transcribe a recorded meeting, a podcast) where you don't care about a few seconds of latency. Streaming sends audio in chunks over a websocket and returns partial transcripts that update live, finalizing as words settle — this is mandatory for anything real-time (live captions, voice agents), because you can't wait for the user to finish a paragraph.
  • Diarization & timestamps. For multi-speaker audio (calls, meetings) you want diarization (label Speaker 1 / Speaker 2) and word timestamps (so you can align, search, and highlight). These are model/feature flags, not free — turn them on only when you need them.
  • VAD — Voice Activity Detection — is your silence guard. This one bites teams hard: when you feed an ASR model silence or non-speech noise (a pause, a fan, hold music), Whisper-class models hallucinate — they invent plausible phantom phrases like ‘thank you for watching’ out of nothing. The fix is VAD: detect where speech actually is and only send those segments to the model, trimming the silence. VAD is also what tells a real-time system when the user stopped talking (endpointing) — which becomes critical for voice agents in L253.

Here's a streaming-friendly transcription with word timestamps (provider-agnostic concept; OpenAI's audio API shown — Claude has no audio endpoint):

from openai import OpenAI
client = OpenAI()

# BATCH: whole file in, full transcript out — for offline work.
with open("call.wav", "rb") as f:
    tr = client.audio.transcriptions.create(
        model="gpt-4o-transcribe",          # or Deepgram Nova / AssemblyAI Universal
        file=f,
        response_format="verbose_json",     # → text + word-level timestamps
        # timestamp_granularities=["word"],
    )
print(tr.text)
for w in tr.words:                          # align / highlight / search the audio
    print(f"{w.start:.2f}-{w.end:.2f}  {w.word}")

# REAL-TIME: open a streaming/websocket session instead, feed mic chunks,
# receive PARTIAL transcripts that finalize — and run VAD first to drop silence
# so the model never hallucinates on non-speech.

STT Failure Modes — Noise, Accents, Homophones, Hallucination

ASR fails in patterned ways. Designing around them is what separates a transcription demo from a system people trust:

  • Noise & far-field audio. Background chatter, a bad mic, a phone on speaker — WER spikes. Mitigate with noise suppression, a good capture path, and VAD; don't expect miracles from a noisy room.
  • Accents & dialects. Models are best on the accents they saw most in training; under-represented accents see much higher WER. Test on your actual users, and prefer models with strong multilingual / accent coverage if that's your audience.
  • Homophones & domain terms. ‘their/there’, ‘to/two/too’, and especially proper nouns, drug names, SKUs, and acronyms get mangled — the model picks the common word, not your rare one. Fix with keyterm / vocabulary prompting (tell the model the special words to expect) and a domain-aware post-correction pass.
  • Hallucination on silence/noise (from the last section) — phantom text on non-speech. VAD is the front-line defense.

The throughline: the transcript is a confident draft, not ground truth — exactly like the vision outputs in L250. For anything high-stakes (a medical note, a legal record, a payment instruction), keep a confidence signal and a human check on the words that matter, and measure WER on real audio rather than trusting the leaderboard.

Text-to-Speech — Voices, Expressiveness, and SSML

The other pillar: turning your model's text reply into natural speech. Modern neural TTS (ElevenLabs, OpenAI, Cartesia, Deepgram Aura, Hume) is genuinely good — the robotic voice of the past is gone. The axes you choose along:

  • Voice & expressiveness. Providers offer libraries of voices (ElevenLabs alone exposes thousands across 70+ languages) ranging from flat-and-fast to emotionally expressive (laughter, emphasis, sighs). Pick by your use case — a meditation app and a drive-through order bot want very different voices.
  • Voice cloning. You can clone a voice from a sample — zero-shot from a few seconds, or a professional clone from 30+ minutes — to give your product a consistent, branded voice. (This is also a consent and safety issue — clone only voices you're authorized to use; deepfake-voice fraud is real.)
  • Control with SSML / audio tags. The model will often mispronounce the things that matter — ‘7pm’, ‘API’, ‘Dr.’, a phone number, a stock ticker — because it guesses from spelling. SSML (Speech Synthesis Markup Language) and newer inline audio tags let you force pronunciation, insert pauses, set rate/pitch/emphasis, and spell things out. Normalize numbers, dates, and acronyms before synthesis, or your polished voice will confidently say ‘seven p-m’ wrong.

Here's streaming TTS — start speaking as bytes arrive, don't wait for the whole clip:

from openai import OpenAI
client = OpenAI()

# STREAM the audio out — play bytes as they arrive (low time-to-first-audio).
with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",                # or ElevenLabs Flash / Cartesia Sonic
    voice="alloy",
    input="Sure — booking a table for two at 7 PM.",   # normalize "7pm" → "7 PM" so it reads right
    response_format="pcm",                  # raw PCM for the lowest latency in a live pipeline
) as resp:
    for chunk in resp.iter_bytes():
        speaker.write(chunk)                # → play immediately; do NOT buffer the whole file

The Latency Game — Time-to-First-Audio and the 300ms Budget

For offline audio, latency is irrelevant — transcribe the podcast overnight. For conversational audio, latency is everything, because humans have a hard-wired expectation for how fast a reply should come. Cross it and the conversation breaks: the user thinks you didn't hear them, starts talking again, and you're stepping on each other.

The number to internalize: a natural turn wants a response in roughly 300 milliseconds, and the whole budget — network transport, STT finalizing, the LLM, and TTS starting — is only a few hundred milliseconds before it feels laggy. You hit that budget with one technique applied everywhere: streaming.

  • Streaming STT gives you a near-final transcript the instant the user stops, instead of waiting for the full clip.
  • Streaming LLM emits its first token in a few hundred ms (TTFT, L224), instead of waiting for the whole reply.
  • Streaming TTS starts producing audio at its time-to-first-audio (TTFA) — the fastest models are ~40–100ms — and crucially, it can begin speaking the first sentence while the LLM is still writing the second.

Chain those and a well-built streaming cascade lands around 400–700ms to first sound — inside the conversational budget. Do the naive thing — batch each stage, wait for the full clip, then the full reply, then the full synthesis — and you're at 2–4 seconds, which is unusable for live talk. Streaming isn't an optimization for voice; it's the difference between a product and a toy — and the interactive lets you flip it off and feel the cliff.

The Architecture Choice — Cascaded vs. Speech-to-Speech

There are two ways to build the voice loop, and it's a real design decision with no universal winner:

Cascaded (STT → LLM → TTS). Three separate services you wire together. The advantages are the reason it dominates production:

  • Control & flexibility — swap any stage independently (5+ STT vendors, dozens of LLMs, 7+ TTS vendors); use the best model for each job.
  • Observability — each stage emits a trace (the transcript, the LLM's text, the audio), so when a call goes wrong you can see exactly which stage failed (L249). You can log and audit the transcript for compliance.
  • Predictable cost (~$0.01–0.17/min) and the ability to run stages in different regions / on-prem.
    The cost: more moving parts, and the summed latency of three hops.

Speech-to-speech (one model: audio in → audio out). Newer native audio models (OpenAI Realtime, Gemini Live) hear and speak directly, with no text round-trip. The advantages:

  • Lowest latency (~250–350ms) and the most natural prosody (it hears how you said something — tone, emotion — which a transcript throws away).
    The costs are real: you lose the transcript and per-stage traces (harder to debug/audit), you're locked to one vendor, and pricing can be far higher and more variable (from fractions of a cent to ~$0.30/min).

Rule of thumb: reach for cascaded when you need control, observability, compliance, or provider choice (most enterprise voice). Reach for speech-to-speech when raw latency and natural prosody are the product and you can accept the lock-in and cost. Either way, the building blocks in this lesson are what you're composing — and L253 turns them into a real, interruptible agent.

See It — The Speech Pipeline Lab

Hold the whole pipeline in your hands. Type what you ‘say’, pick the audio condition (clean / accent / noisy), toggle streaming and the architecture (cascaded vs speech-to-speech), and choose a TTS voice. Watch the transcript degrade, the per-stage latency add up to a time-to-first-sound, and a verdict against the conversational budget.

The Speech Pipeline Lab — type what you 'say', pick the audio condition (clean / accent / noisy), toggle streaming and the architecture (cascaded STT→LLM→TTS vs a native speech-to-speech model), and choose a TTS voice. Watch the transcript degrade (rising WER, a Whisper-style hallucination on noisy audio), the per-stage latency add up to a time-to-first-sound, and a verdict against the conversational budget. Turn streaming off to feel why batch is unusable for live talk; switch to speech-to-speech to see latency drop and observability disappear.

Three things to try:

  1. Break the transcript. Switch the condition to noisy and watch WER jump and a phantom ‘…thank you for watching’ appear — a real Whisper hallucination on non-speech. That's why you run VAD first.
  2. Feel the streaming cliff. On cascaded, toggle Streaming OFF: time-to-first-sound jumps from ~700ms to seconds and the verdict flips to laggy. Streaming is the whole latency game.
  3. Weigh the architectures. Flip to Speech-to-speech: latency drops to ~300ms (snappiest) — but notice the cost jumps to ~$0.30/min and you'd lose the transcript and per-stage traces. That's the cascaded-vs-S2S trade, made concrete.

The takeaway in your hands: STT in, TTS out; measure the transcript with WER; protect it with VAD; and stream every stage to live inside the conversational budget.

🧪 Try It Yourself

Reason it through (then check in the lab):

  1. Your transcription service reports 6% WER on the vendor benchmark, but in production your call-center transcripts are full of errors. Give two likely reasons and two fixes.
  2. A user pauses for two seconds mid-sentence and your transcript suddenly contains ‘thank you, goodbye’ that they never said. What happened, and what's the fix?
  3. Your voice agent's TTS keeps saying customers' order totals wrong — ‘$1,500’ comes out as ‘fifteen hundred… dollars sign’. What do you change?
  4. You need the lowest possible latency for a consumer voice chat and don't need transcripts or compliance logging. Cascaded or speech-to-speech — and what do you give up?

Answers:

  1. Your audio isn't the benchmark's: noise/far-field mics and accents/domain jargon drive WER up. Fixes: measure WER on your own audio, add noise suppression + VAD, use keyterm / vocabulary prompting for your names and numbers, and pick a model strong on your users' accents.
  2. Hallucination on silence — the ASR model invented text from the non-speech pause. Fix: run VAD to detect and send only actual speech, trimming the silence so the model never sees non-speech.
  3. Text normalization + SSML. Convert ‘$1,500’ to spoken form (‘one thousand five hundred dollars’) before synthesis, and use SSML/audio tags to force pronunciation of numbers, currency, and acronyms. The TTS guesses from spelling unless you tell it.
  4. Speech-to-speech — lowest latency (~300ms) and natural prosody. You give up the transcript and per-stage traces (harder to debug/audit), provider flexibility (vendor lock-in), and often pay more per minute. Fine here because you don't need the transcript or compliance trail.

Mental-Model Corrections

  • “6% WER means it's basically perfect.” → On clean audio. On your noisy, accented, jargon-heavy audio it can be 20–30%+. Measure on your data, and weight the words that matter (names, numbers).
  • “Silence transcribes as nothing.” → No — ASR models hallucinate phantom phrases on silence and noise. Run VAD to send only real speech.
  • “TTS just reads the text.” → It guesses pronunciation and gets numbers, dates, currency, and acronyms wrong. Normalize the text and use SSML.
  • “Make the LLM faster to fix voice latency.” → The LLM is one of three hops. The real lever is streaming every stage (STT, LLM, TTS) so audio starts before each stage finishes — that's how you hit the ~300ms budget.
  • “Batch is fine, just make it quick.” → Batch waits for full clips and is seconds to first sound — unusable for live conversation. Batch is for offline transcription; live needs streaming.
  • “Speech-to-speech is strictly better — it's faster.” → It's lower-latency, but you lose the transcript, per-stage traces, and provider choice, and it can cost far more. Cascaded still wins for control, observability, and compliance.
  • “Claude can transcribe my audio.” → Claude is text + vision + documents, no native audio I/O. Pair it with a specialized STT/TTS provider in a cascaded pipeline.

Key Takeaways

  • Audio = two pillars: STT (audio → text) and TTS (text → audio), chained with an LLM into the voice loop 🎤 → STT → LLM → TTS → 🔊.

  • Measure STT with WER = (subs + dels + ins) / words; SOTA is ~5–6% on clean audio but far worse on noise/accents — evaluate on your own audio, weight the words that matter.

  • STT practicals: stream for real-time (batch for offline), use diarization + timestamps when needed, and run VAD to strip silence so the model doesn't hallucinate phantom text.

  • TTS: choose voice + expressiveness, watch for mispronounced numbers/acronyms (normalize + SSML), and optimize time-to-first-audio; voice cloning is powerful and a consent issue.

  • The latency game: conversation wants a reply in ~300ms, so stream every stage — a good streaming cascade hits ~400–700ms to first sound; batch (seconds) is unusable for live talk.

  • Architecture: cascaded (control, observability, provider choice, predictable cost) vs speech-to-speech (lowest latency + natural prosody, but lock-in, no transcript/traces, pricier).

  • Claude has no native audio — pair it with specialized STT/TTS providers.

  • Next — L253: Building Voice Agents. You have the building blocks; next you assemble them into a live, interruptible voice agent — turn-taking, barge-in, endpointing, and dialogue management — the hardest and most magical part of conversational AI.