Advanced Systems
Production speech products need more than WER. This lesson connects ASR, TTS, speech-to-speech, RAG, human review, and online monitoring into one evaluation program an experienced engineer can defend.
Evaluation Stack
Audio-text systems fail across multiple boundaries. Keep separate metrics for the acoustic model, text model, retrieval layer, response generator, speech renderer, and end-to-end user task.
Track WER, CER, entity error rate, timestamp error, diarization error, partial churn, endpointing delay, and language ID accuracy.
WER weights every word equally. Users may care more about names, numbers, product codes, commands, timestamps, punctuation, and diarization. A model can reduce filler-word errors while making more business-critical entity mistakes.
Track intelligibility, MOS or preference win rate, first audio byte, streaming stability, speaker similarity, prosody, safety, and interruption behavior.
Block release if intelligibility drops, first-audio-byte exceeds the conversational SLO, voices become unstable, unsafe text is spoken, speaker identity controls regress, or a quality gain only appears on clean lab prompts but fails on real product text.
Track task completion, correction rate, barge-in success, turn latency, hallucination rate, refusal quality, and recovery from interrupted speech.
End-to-end metrics tell whether users succeed, but they are hard to debug alone. Component metrics localize the failure. If task completion drops, separate ASR errors, retrieval misses, reasoning failures, TTS latency, and UI playback problems.
Report metrics by accent, noise, channel, microphone, language, domain terms, speaking rate, long silence, overlap, and device class.
A slice is launch-critical when it maps to a major user group, safety risk, contractual requirement, revenue workflow, or known model weakness. Experienced engineers protect these slices with explicit regression budgets and rollback criteria.
RAG For Speech Products
Voice assistants often answer from docs, tickets, transcripts, or user memory. Retrieval improves grounding, but ASR errors and spoken ambiguity make retrieval evaluation different from text-only RAG.
A customer asks spoken questions about a product manual. The system must answer by voice, cite the manual internally, and avoid exposing private transcript text in logs.
Use streaming ASR with confidence and timing metadata, normalize product terms, retrieve against chunked manual passages, rerank with query and conversation context, generate a grounded answer, and synthesize speech. Evaluate ASR entity accuracy, retrieval recall at k, answer faithfulness, citation support, latency, and redaction. Log trace IDs, versions, aggregate metrics, and privacy-safe error categories rather than raw private audio by default.
Users ask for "XR-17 battery calibration", but ASR often returns "ex are seventeen battery celebration". Retrieval misses the right manual page. What do you change?
Add domain vocabulary biasing, pronunciation variants, entity normalization, fuzzy lexical retrieval, character n-gram features, audio-confidence-aware query rewriting, and retrieval evaluation with noisy ASR hypotheses. Keep a fallback clarification question when confidence is low or top passages disagree.
Judges
Classify answer helpfulness, detect unsupported claims, summarize error reports, and prioritize human review queues after calibration.
Build a labeled set, compare judge decisions to human labels, inspect disagreements, measure precision and recall by slice, lock the judge prompt and model version, and avoid using the judge as the only launch gate for high-risk behavior.
Replacing human labels for safety-critical cases, judging private content without retention controls, or using a judge that shares the same blind spots as the generator.
Log judge model version, prompt version, rubric, score, reason category, request trace ID, and redacted inputs when allowed. Keep enough metadata to reproduce decisions without making the monitoring system a private-data sink.
Incident Drills
Offline faithfulness improved after a retriever update, but users report that spoken answers are less useful.
Check whether the eval set overrepresents written queries instead of ASR transcripts, whether retrieved chunks are too long for spoken answers, whether citations are correct but answers omit the user's actual intent, and whether TTS latency or prosody makes good text feel bad. Compare query ASR, retrieval recall, generator answer, TTS output, and user task completion on the same traces.
A new monitoring rule sends too many voice-agent sessions to human review. Accuracy is better, but the operations team is overloaded.
Measure alert precision, duplicate alerts per session, slice concentration, confidence thresholds, and whether a cheap automatic classifier can triage low-risk cases. Keep high-risk review paths, but tune routing, batch similar failures, sample for trend detection, and report the cost-quality tradeoff explicitly.
Lab
Create a small, privacy-safe evaluation table for a voice RAG system. Use synthetic or sanitized examples only.
It connects offline model metrics to production decisions. An experienced engineer does not stop at "the model improved"; they define the user task, measure each system boundary, protect private data, price the review burden, and state what would block rollout.