Production Track
Practice the first hour of debugging for ASR, TTS, speech-to-speech, and audio retrieval systems. The goal is to connect model behavior, serving systems, telemetry, privacy, and rollback judgment.
Operating Loop
Experienced engineers are judged by how quickly they separate product impact, model quality, data drift, serving behavior, and release safety. Use this loop for incident interviews and real launches.
Speech systems can fail before the model sees useful features: microphone path, sample rate, channel layout, clipping, VAD, endpointing, codec artifacts, packet jitter, echo, playback timing, and partial-result UI can all create user-visible failures. A strong answer debugs the full audio path, not only model weights.
Telemetry
Queue age, batch size, timeout count, retry count, GPU memory, CPU preprocess time, cold starts, autoscaler lag, and fallback rate.
Compare the same signals by model version and traffic cohort. A p99 spike with unchanged model time points at queues, routing, autoscaling, batching, retries, or dependency calls. A model-time spike points at longer inputs, larger beams, precision changes, cache misses, device placement, or a new model artifact.
Duration, sample rate, channel count, clipping ratio, silence ratio, VAD segment count, language probability, SNR proxy, and confidence.
Prefer aggregate feature summaries, redacted transcripts, synthetic canaries, public fixtures, and explicit opt-in review. Raw private audio should not be the default debugging artifact. Capture enough derived evidence to isolate drift without creating a new privacy problem.
Incident Practice
Treat these as interview prompts. State assumptions, name the likely failure planes, and finish with mitigation and prevention.
Offline WER has not regressed, but users say the live dictation experience became less trustworthy after a streaming decoder update.
WER can miss partial churn, endpoint delay, punctuation flicker, timestamp jumps, capitalization edits, UI commit behavior, and domain entity instability. Compare partial-hypothesis churn, first partial latency, finalization delay, edit distance over time, and user correction rate. Roll back if live trust metrics regressed even when final WER looks acceptable.
A new voice improves offline preference tests, but conversation turns feel laggy and users interrupt more often.
Break down text normalization, sentence segmentation, acoustic model latency, vocoder latency, first audio byte, streaming chunk cadence, playback buffer, and client scheduling. Preference score is not enough for conversational quality. Mitigate with sentence streaming, warm pools, shorter first chunks, voice fallback, lower-cost vocoding for long tails, or rollback.
Text queries still retrieve the right documents, but spoken questions now produce weaker answers in the voice assistant.
Inspect ASR substitutions on named entities, punctuation and casing changes, query rewrite prompts, embedding model version, chunking, top-k, filters, language detection, and answer-grounding policy. Build paired evals from clean text, ASR hypotheses, and noisy ASR hypotheses so retrieval regressions are caught before launch.
You rolled back the model version, but p99 latency remains high. What do you investigate next?
Check queues left behind by the bad release, autoscaler state, warmed model pools, cache fragmentation, client retry storms, shadow traffic, changed feature flags, dependency rate limits, schema migrations, and stuck workers. A rollback plan should include draining queues, clearing bad caches, disabling shadow traffic, and verifying recovery with synthetic probes.
Coding Lab
These examples are intentionally simple. They train interview-ready thinking: define the signal, compute it reliably, and know the edge cases.
Given a stream of interim ASR hypotheses, compute how often the text changes after the previous hypothesis.
Invariant: each step compares one hypothesis with the immediately preceding hypothesis. Empty streams have zero churn, and repeated identical partials should not inflate the metric.
def partial_churn_rate(partials):
if len(partials) < 2:
return 0.0
changes = 0
for prev, current in zip(partials, partials[1:]):
if prev != current:
changes += 1
return changes / (len(partials) - 1)Given request records with audio duration and latency, compute average latency for short, medium, and long audio.
Invariant: every record is assigned to exactly one duration bucket.
Missing buckets return None rather than pretending the
average is zero.
def latency_by_duration(records):
buckets = {
"short": [],
"medium": [],
"long": [],
}
for item in records:
duration = item["audio_seconds"]
latency = item["latency_ms"]
if duration < 5:
buckets["short"].append(latency)
elif duration < 30:
buckets["medium"].append(latency)
else:
buckets["long"].append(latency)
return {
name: (sum(values) / len(values) if values else None)
for name, values in buckets.items()
}Exam Prompts
The team wants to ship a faster streaming ASR model. What launch gate would you require?
Include global WER plus slice WER, entity error, partial churn, first partial latency, finalization delay, p95 and p99 server latency, GPU cost per audio minute, privacy review, canary plan, rollback switch, and owner. The gate should define explicit regression budgets, not only "looks good" review.
A VAD threshold change caused dropped first words for quiet speakers. How do you hotfix without creating a second outage?
Restore the prior threshold for affected cohorts or roll back the feature flag first. Reproduce with quiet-speaker fixtures, add a regression test for leading-word deletion, canary the corrected threshold, monitor first-token deletion, latency, false starts, and VAD segment count, then update the release checklist.