Module 21
ML engineers are expected to turn ASR, TTS, LLM, and speech-to-speech models into reliable products. This lesson practices readiness reviews: capacity, rollout gates, monitoring, incident response, and cost-aware serving decisions.
Architecture Review
A production speech platform usually serves several workloads at once: wake word, VAD, streaming ASR, turn classification, LLM reasoning, retrieval, TTS, evaluation jobs, and offline transcription. Strong design starts by separating user-visible latency paths from throughput paths.
Optimize first partial transcript, first spoken response, queue age, jitter, cancellation, barge-in, and fallback behavior.
Offline transcription, eval backfills, embedding generation, batch redaction, shadow experiments, and heavy model sweeps. These jobs can consume GPU memory and scheduler attention without improving live user experience.
Optimize accelerator utilization, idempotent writes, checkpointing, retry safety, lineage, and reproducible model or prompt versions.
It records input manifest versions, model hashes, decoding settings, prompt versions, redaction policy, output schema, checkpoint cursor, retry count, and aggregate quality metrics. It should never require committing private audio or transcripts.
Launch Gates
A new ASR model improves average WER by 4 percent relative, but noisy far-field traffic regresses by 8 percent relative. p95 first partial transcript latency is unchanged. Should you launch?
Do not launch broadly. A strong answer protects the regressed slice, checks whether far-field traffic is strategically important, audits sample size and label quality, and proposes a constrained canary or model routing rule only if the affected slice can be excluded. The launch gate should include slice-level WER, entity recall, latency, privacy logging checks, and rollback triggers.
A new TTS voice has better MOS in human review, but first-audio-byte p99 is 700 ms slower and GPU cost per minute is 35 percent higher. Product wants to ship before a demo.
Ship only behind an explicit product decision and a narrow audience if the demo value justifies it. Keep the old voice as fallback, enforce p99 and cost alerts, pre-warm where needed, and define a rollback threshold. A strong strong answer separates subjective quality wins from reliability and unit-economics risks.
Retrieval recall improved, but spoken answers now cite stale documents more often because the top-k includes older policy pages.
Add freshness-aware ranking, metadata filters, stale-source evaluation slices, and answer-level citation checks. Launch gates should measure retrieval recall, source freshness, groundedness, refusal behavior, ASR-noise robustness, and user-visible latency.
Incident Practice
Users complain that live captions keep changing. Final WER is stable, but partial text feels unusable.
Inspect partial rewrite rate, endpointing thresholds, VAD sensitivity, beam stability, token timestamps, noise slices, client debounce behavior, and recent decoder changes. Mitigate by stabilizing committed prefixes, tuning endpointing, or rolling back the streaming decoder.
Daily spend doubled with no obvious traffic growth. Quality metrics are flat and user-visible latency is slightly better.
Check batch size, utilization, autoscaler floor, shadow traffic, eval jobs running on live pools, retry loops, longer contexts, disabled quantization, over-warm replicas, and tenant mix. The fix should preserve SLOs while restoring utilization and routing batch work away from live capacity.
One region reports worse command recognition after a deployment. Global dashboards look normal.
Slice by region, language, accent, microphone path, network jitter, client version, model version, feature extraction version, and routing policy. Compare canary and control cohorts. An experienced responder does not rely on global WER during a regional complaint.
Abuse monitoring detects a burst of synthetic voice requests that resemble impersonation attempts.
Rate-limit suspicious tenants, preserve privacy-safe aggregate evidence, disable risky voices if policy allows, inspect prompt and account patterns, escalate to trust and safety, and confirm that logs do not expose private voiceprints or raw recordings.
Coding Lab
Implement small utilities that mirror production review work. Use synthetic aggregate metrics only.
Given metric deltas for a model canary, decide whether to continue, pause, or rollback. Negative WER and latency deltas are improvements.
def rollout_decision(metrics):
hard_failures = []
if metrics["p95_latency_ms_delta"] > 100:
hard_failures.append("latency")
if metrics["far_field_wer_relative_delta"] > 0.03:
hard_failures.append("far_field_wer")
if metrics["privacy_error_count"] > 0:
hard_failures.append("privacy")
if hard_failures:
return {"decision": "rollback", "reasons": hard_failures}
if metrics["overall_wer_relative_delta"] <= -0.02:
return {"decision": "continue", "reasons": ["quality_gain"]}
return {"decision": "pause", "reasons": ["insufficient_gain"]}The invariant is that privacy errors and protected-slice regressions dominate average quality wins. A common mistake is launching because aggregate WER improved.
Compute the burn rate for a queue latency SLO. The error budget is the allowed fraction of requests above threshold.
def queue_burn_rate(latencies_ms, threshold_ms=250, allowed_bad_fraction=0.01):
if not latencies_ms:
return 0.0
bad = sum(x > threshold_ms for x in latencies_ms)
observed_bad_fraction = bad / len(latencies_ms)
return observed_bad_fraction / allowed_bad_fractionA burn rate above 1 means the service is consuming error budget faster than planned for that window. Alerting should combine short and long windows to catch both spikes and slow burns.
Compare old and new cost per successful speech turn. Return the top drivers that need investigation.
def cost_regression(old, new, tolerance=0.10):
old_unit = old["gpu_dollars"] / max(old["successful_turns"], 1)
new_unit = new["gpu_dollars"] / max(new["successful_turns"], 1)
relative = (new_unit - old_unit) / old_unit
drivers = []
for key in ("avg_context_tokens", "retry_rate", "shadow_traffic_fraction"):
if new[key] > old[key]:
drivers.append(key)
return {
"old_unit_cost": old_unit,
"new_unit_cost": new_unit,
"relative_delta": relative,
"regressed": relative > tolerance,
"drivers": drivers,
}The useful metric is cost per successful user outcome, not total spend alone. Rising traffic may be acceptable; rising unit cost requires a capacity or product explanation.
Advanced Practice Prompts
Design a platform that hosts streaming ASR, TTS, and speech-to-speech for multiple product teams. Cover APIs, tenancy, scheduling, observability, release gates, privacy, and cost controls.
Discuss separate real-time and batch pools, tenant quotas, model registry, feature extraction contracts, streaming APIs, cancellation, canary routing, slice metrics, privacy-safe telemetry, incident runbooks, model rollback, autoscaling, GPU utilization, and chargeback or showback. Call out that ASR, TTS, and LLM serving have different bottlenecks.
p95 turn latency increased by 35 percent after a release. ASR, LLM, and TTS each claim their local metrics are healthy.
Build an end-to-end trace waterfall and inspect queue time, network, client buffering, orchestration gaps, retries, retrieval latency, context growth, TTS first-audio-byte, playback start, and cancellation. Local component health is not enough when the product SLO is a full spoken turn.
A privacy-sensitive assistant needs wake word, short commands, long dictation, and high-quality TTS. Decide what runs locally versus in the cloud.
Run wake word, VAD, privacy filters, and simple commands locally when possible. Use cloud or dedicated servers for long dictation, large LLM reasoning, heavy retrieval, and high-quality TTS if consent and policy allow. Add offline fallback, explicit upload boundaries, telemetry minimization, and model/version compatibility checks.