Production ML Track
Turn ASR, TTS, speech-to-speech, and audio RAG systems into services you can launch, monitor, debug, and roll back with advanced discipline.
Service Level Thinking
A speech system can look healthy on average while the conversation feels broken. Experienced engineers translate user pain into measurable quality, latency, availability, privacy, and cost budgets.
Track first partial latency, partial churn, finalization delay, WER slices, entity errors, timeout rate, and cost per audio minute.
Require no critical slice WER regression, bounded entity error, p95 first partial under the product target, p99 finalization under the SLO, no increase in privacy-sensitive logging, and a versioned rollback switch. Aggregate WER alone is not enough.
Track first audio byte, chunk cadence, underruns, MOS or preference score, interruption rate, abandonment, and cost per generated second.
A better voice can still be a worse product if it delays the first audible response. Use a two-tier gate: conversational latency and playback stability must pass before preference wins can justify a rollout.
Dashboards
Dashboards should answer whether to continue, pause, roll back, route around a dependency, or start a data investigation.
Model metrics explain quality regressions; serving metrics explain resource and latency behavior. Mixing them hides root cause. For example, worse p99 latency with stable model time points at queues, batching, routing, autoscaling, dependencies, or retries.
Coding Labs
These snippets are intentionally interview-sized. Each one trains the habit of turning vague operational symptoms into testable signals.
Given request counts and bad request counts in a window, compute error-budget burn rate relative to an allowed bad-request fraction.
Invariant: burn rate compares observed bad fraction to the allowed bad fraction. Test zero requests, no errors, exactly-on-budget errors, and a window that burns several times faster than budget.
def burn_rate(total_requests, bad_requests, allowed_bad_fraction):
if total_requests == 0:
return 0.0
if allowed_bad_fraction <= 0:
raise ValueError("allowed_bad_fraction must be positive")
observed = bad_requests / total_requests
return observed / allowed_bad_fractionGiven baseline and current aggregate feature summaries for audio slices, return slices whose relative change exceeds a threshold.
Invariant: compare each slice only when both baseline and current values exist. Treat a zero baseline as a special case so the code does not hide a new population. Test missing slices, zero baseline, small changes, and large increases in silence or clipping.
def drifted_slices(baseline, current, threshold):
drifted = []
for name, now in current.items():
if name not in baseline:
drifted.append((name, "new_slice", now))
continue
before = baseline[name]
if before == 0:
if now != 0:
drifted.append((name, "zero_baseline", now))
continue
relative = abs(now - before) / abs(before)
if relative > threshold:
drifted.append((name, "relative_change", relative))
return driftedGiven canary metrics and budgets, return whether a speech model rollout should continue, pause, or roll back.
Invariant: critical metrics override noncritical wins. Test a latency win with entity error regression, missing critical metrics, exactly-on-budget deltas, and a noncritical cost regression that should pause instead of roll back.
def rollout_recommendation(metrics, budgets):
findings = []
for name, budget in budgets.items():
if name not in metrics:
findings.append(("pause", name, "missing"))
continue
value = metrics[name]
limit = budget["limit"]
direction = budget["direction"]
critical = budget.get("critical", False)
failed = (
direction == "lower_is_better" and value > limit
) or (
direction == "higher_is_better" and value < limit
)
if failed:
action = "rollback" if critical else "pause"
findings.append((action, name, value))
if any(item[0] == "rollback" for item in findings):
return "rollback", findings
if findings:
return "pause", findings
return "continue", []Interview Prompts
The new ASR model improves aggregate WER by 4 percent, but a noisy mobile cohort in one region regresses by 11 percent. What do you do?
Do not hide behind the aggregate win. Pause or exclude the affected cohort, verify sample size and label quality, inspect audio feature drift, compare entity error and correction rate, and define a slice gate before resuming. A strong answer protects harmed cohorts while preserving the ability to ship where the model is truly better.
ASR quality and latency are stable, but GPU cost per audio minute doubles after a release. How do you investigate?
Check model artifact size, precision, batch shape, request mix, VAD savings, retries, shadow traffic, autoscaler target, warm pool size, GPU utilization, cache behavior, and route weights. Mitigate with quota, route rollback, batch tuning, disabling shadow traffic, or reverting precision/config changes while preserving quality.
An endpointing bug clipped the first word for quiet speakers for three hours. What should the postmortem contain?
Include timeline, detection gap, user impact by slice, root cause, why tests missed quiet speakers, rollback timeline, privacy-safe evidence, immediate remediation, new synthetic fixtures, release gate updates, owners, deadlines, and a follow-up check that proves the prevention work actually shipped.