Advanced System Design
Learn to design audio-text ML systems like an experienced engineer: clarify product goals, decompose services, choose model and serving paths, define launch gates, and defend rollback, observability, cost, and privacy.
Interview Framework
A strong system design answer begins by shaping the problem. Speech systems are sensitive to latency, audio quality, privacy, streaming behavior, domain vocabulary, device differences, and rollout risk.
Jumping straight to model names. A strong answer first exposes the product and operational constraints, then picks architecture. The right model depends on whether the system needs streaming partials, exact transcripts, natural turn-taking, private local inference, cheap batch throughput, multilingual support, or low first-audio-byte latency.
Reference Architecture
Use a cascaded design as the default baseline because it is easier to evaluate, monitor, debug, and roll back than a fully direct model.
Client capture, echo cancellation, VAD, streaming upload, ASR partials, turn manager, LLM or tool planner, TTS streamer, playback, and telemetry.
Preserve trace ID, model versions, audio clock, chunk index, endpoint decisions, partial transcript history, final transcript, prompt version, retrieved document IDs, TTS voice version, and client playback events. This lets you debug a bad turn without storing raw private audio by default.
Sanitized eval fixtures, synthetic canaries, opted-in review queues, batch scoring, model registry, release notes, and regression dashboards.
Offline evals give repeatable comparisons on known slices. Online metrics catch distribution shifts, UI timing failures, retry storms, device regressions, and subjective experience changes such as partial churn or slow first audio. Launch decisions need both.
Tradeoffs
Budget capture, VAD, first partial, final transcript, LLM planning, first audio byte, chunk cadence, and playback buffer.
Give a budget before proposing optimization. For example: 150 ms client buffering, 300 ms first partial, 800 ms finalization for short turns, 500 ms planning, and 400 ms first audio byte. Then explain which stages can stream and which are blocking.
Control GPU time with model routing, quantization, batching, short utterance fast paths, warm pool sizing, and shadow traffic limits.
Estimate cost from audio minutes, model real-time factor, hardware hourly price, utilization, retry rate, and duplicate traffic. A 1.0x real-time ASR model at 50 percent utilization costs roughly twice as much per audio minute as the same model at full utilization before accounting for failed requests.
Track WER, entity error, turn success, groundedness, TTS preference, pronunciation, safety, and correction rate by slice.
Aggregate WER can hide failures on names, addresses, rare domain terms, accents, noisy microphones, code-switching, quiet speakers, long-form dictation, and streaming partial stability. Slice metrics and product metrics prevent a launch that looks good only on average.
Production Exercises
A quantized ASR model lowers p50 latency by 30 percent, but user text corrections rise in the canary.
Compare correction rate by entity-heavy turns, noise level, language, device, utterance length, and confidence. Inspect WER, entity error, timestamp drift, decoder beam settings, punctuation, and postprocessing. Keep the latency win only if the regression is within the launch budget or can be routed away from sensitive cohorts. Otherwise pause or roll back.
Users keep interrupting the assistant after a voice rollback. The voice model is back to the old version, but interruption rate remains high.
Look beyond the voice model: sentence segmentation, LLM response length, first audio byte, playback buffer, client audio focus, echo cancellation, barge-in detector threshold, turn manager state, and cached responses. Rollback may need feature flags, cache invalidation, queue draining, and client config restoration.
Exam Prompts
Design a dictation service for professionals who need low latency, domain vocabulary, privacy controls, and reliable edits.
Clarify latency and accuracy SLOs, supported devices, vocabulary update path, retention rules, and offline fallback. Propose client capture plus VAD, streaming ASR with partial stabilization, custom vocabulary or contextual biasing, punctuation restoration, edit history, privacy-safe traces, model registry, eval slices for domain terms, canary rollout, rollback, and dashboards for partial latency, finalization, correction rate, entity error, and cost per audio minute.
Users ask spoken questions over an internal knowledge base. Design the system and evaluation plan.
Use ASR, query rewrite, retrieval, reranking, grounded generation, citation policy, TTS, and feedback collection. Evaluate clean text queries, ASR hypotheses, noisy ASR hypotheses, retrieval recall, answer groundedness, refusal correctness, first audio byte, and end-to-end task success. Monitor ASR entity substitutions because one wrong name can break retrieval even when transcript WER looks acceptable.
A research team proposes replacing the cascaded assistant with a direct speech-to-speech model. What is your review?
Compare latency, natural prosody, controllability, transcript auditability, safety filters, retrieval integration, tool use, observability, eval maturity, data requirements, rollback, and cost. A good recommendation may be hybrid: use direct speech for low-risk conversational responses while keeping cascaded paths for tool use, regulated content, debugging, and high-precision tasks.
Coding Lab
Small production utilities help in interviews because they force clear definitions and edge-case handling.
Given speech assistant turn records, return the trace IDs where any stage exceeds its budget.
Invariant: each reported trace has at least one named stage whose observed latency is greater than the allowed budget. Missing stages are ignored here because a separate schema validator should catch malformed records.
def budget_violations(records, budgets_ms):
violations = []
for record in records:
stages = record.get("stages_ms", {})
bad = {
name: value
for name, value in stages.items()
if name in budgets_ms and value > budgets_ms[name]
}
if bad:
violations.append({
"trace_id": record["trace_id"],
"violations": bad,
})
return violations