Production Practice
Learn how ML engineers prove that ASR, TTS, retrieval, LLM, and speech-to-speech systems will keep working when traffic, latency, dependency failures, and cost pressure arrive at the same time.
Operating Model
A speech product can pass unit tests and offline evals while failing live because the real bottleneck is queueing, turn-taking, GPU memory, cache misses, retrieval fanout, TTS warmup, client playback, or a bad retry policy. A strong load test validates quality, latency, cost, and graceful degradation together.
WER only measures transcript text quality on a fixed set. It misses tail latency, partial transcript churn, endpointing errors, GPU queue saturation, retry storms, cost spikes, stale retrieval, TTS first audio byte, and user-visible recovery behavior. Launch readiness needs offline quality plus live-system stress signals.
Load Testing
Good load tests are not random traffic floods. They encode the system invariants that must stay true while traffic changes.
Interactive traffic must not wait behind batch transcription, and first partial p95 must stay below the product target.
Run mixed traffic with short live calls, long batch files, noisy mobile audio, and tenant-specific vocabulary updates. Verify priority queues, per-stream state limits, GPU queue depth, endpointing stability, and fallback to a smaller model when the interactive queue burns error budget.
First audio byte must stay low even when a new voice, long prompt, or cold region enters the serving path.
Test short confirmations, long policy explanations, multilingual text normalization, warm-pool depletion, cache misses, and vocoder placement. Separate first audio byte, full synthesis time, and playback underruns so mitigation is precise.
Retrieval freshness, ACLs, and grounded answers must hold under index refresh and ASR entity errors.
Replay synthetic noisy queries across tenants while rotating index versions. Track retrieval recall, stale-document rate, forbidden document exposure, answer grounding, LLM token cost, and handoff rate. Fail closed when ACL or freshness checks are uncertain.
Cost per successful turn must not exceed the launch budget without a measured quality gain.
Stress long conversations, retrieval fanout, disabled cache, retries, high-temperature regeneration, and fallback chains. Watch tokens per turn, GPU seconds, cache hit rate, successful task completion, and the point where a cheaper route should take over.
Chaos Readiness
Run chaos drills with an explicit owner, small blast radius, production-safe fixtures, customer-impact abort criteria, and a rollback path. Do not inject failures into live customer traffic unless the service has approved guardrails and monitoring in place.
During peak call volume, one serving region loses half of its GPU workers. Batch ASR and interactive voice-agent calls share the same cluster.
Preserve interactive traffic first. Shed or defer batch work, lower max batch delay for live calls, route overflow to a warm fallback region, and downgrade noncritical paths to smaller models. Watch p95 first partial, queue depth, retry rate, GPU memory headroom, tenant error budgets, and cost. The prevention item is hard isolation or strict priority plus a load test proving it.
A release changes text normalization. Cache hit rate falls from 62 percent to 7 percent, first audio byte doubles, and cost rises.
Compare normalized text keys, voice IDs, locale, punctuation, segmentation, and cache namespaces between old and new releases. Roll back if first audio byte or cost-per-success crosses the gate. Prevention: add a CI fixture that snapshots normalization keys for common utterances and a canary gate for cache hit rate.
A vector database upgrade increases p95 retrieval latency. ASR and TTS are healthy, but voice-agent turns feel slow.
Use retrieval timeouts, smaller top-k, cached tenant policy snippets, answer templates for safe intents, or human handoff for high-risk intents. Do not silently answer from stale documents unless the product explicitly allows it. Add dependency-latency chaos tests and grounded-answer checks under timeout behavior.
Coding Labs
These labs use synthetic aggregate metrics. They are safe to commit and useful in interviews because they turn operational judgment into clear, testable rules.
Given aggregate load-test metrics by route, decide whether the release can ship. Each route has p95 latency, error rate, cost per successful turn, and task success delta versus baseline.
Invariant: a route can only pass if user experience, reliability, and cost are all inside budget. Test a clean pass, a latency fail, a cost fail with quality gain, a missing required route, malformed metrics, and an unexpected route without a launch budget.
from collections.abc import Mapping
from math import isfinite
def _mapping(value, field):
if not isinstance(value, Mapping):
raise ValueError(f"{field} must be a mapping")
return value
def _required(mapping, key, field):
if key not in mapping:
raise ValueError(f"{field}.{key} is required")
return mapping[key]
def _finite_non_negative(value, field):
if isinstance(value, bool) or not isinstance(value, (int, float)) or not isfinite(value) or value < 0:
raise ValueError(f"{field} must be a finite non-negative number")
return value
def _finite_number(value, field):
if isinstance(value, bool) or not isinstance(value, (int, float)) or not isfinite(value):
raise ValueError(f"{field} must be a finite number")
return value
def _finite_rate(value, field):
value = _finite_non_negative(value, field)
if value > 1:
raise ValueError(f"{field} must be between 0 and 1")
return value
def evaluate_load_gate(routes, budgets):
routes = _mapping(routes, "routes")
budgets = _mapping(budgets, "budgets")
failures = []
for name in budgets:
if name not in routes:
failures.append((name, "missing_metrics", None))
for name, metrics in routes.items():
if name not in budgets:
failures.append((name, "missing_budget", None))
continue
metrics = _mapping(metrics, name)
budget = budgets[name]
budget = _mapping(budget, f"{name}.budget")
p95_budget_ms = _finite_non_negative(_required(budget, "p95_ms", f"{name}.budget"), f"{name}.budget.p95_ms")
error_budget = _finite_rate(_required(budget, "error_rate", f"{name}.budget"), f"{name}.budget.error_rate")
cost_budget = _finite_non_negative(_required(budget, "cost_per_success", f"{name}.budget"), f"{name}.budget.cost_per_success")
p95_ms = _finite_non_negative(_required(metrics, "p95_ms", name), f"{name}.p95_ms")
error_rate = _finite_rate(_required(metrics, "error_rate", name), f"{name}.error_rate")
cost_per_success = _finite_non_negative(_required(metrics, "cost_per_success", name), f"{name}.cost_per_success")
task_success_delta = _finite_number(_required(metrics, "task_success_delta", name), f"{name}.task_success_delta")
if p95_ms > p95_budget_ms:
failures.append((name, "latency", p95_ms))
if error_rate > error_budget:
failures.append((name, "errors", error_rate))
cost_over = cost_per_success > cost_budget
min_gain = _finite_number(budget.get("min_gain_for_cost_overrun", 0.02), f"{name}.budget.min_gain_for_cost_overrun")
quality_gain = task_success_delta >= min_gain
if cost_over and not quality_gain:
failures.append((name, "cost_without_quality_gain", cost_per_success))
return {"ship": not failures, "failures": failures}Detect when client or service retries are amplifying a dependency failure during a voice-agent incident.
Common mistakes are alerting on request count alone, ignoring the baseline, accepting malformed counters, and missing the dependency error signal. Retry storms show more attempts per original user turn plus rising dependency errors.
from collections.abc import Mapping
from math import isfinite
def _mapping(value, field):
if not isinstance(value, Mapping):
raise ValueError(f"{field} must be a mapping")
return value
def _required(mapping, key, field):
if key not in mapping:
raise ValueError(f"{field}.{key} is required")
return mapping[key]
def _finite_non_negative(value, field):
if isinstance(value, bool) or not isinstance(value, (int, float)) or not isfinite(value) or value < 0:
raise ValueError(f"{field} must be a finite non-negative number")
return value
def _finite_rate(value, field):
value = _finite_non_negative(value, field)
if value > 1:
raise ValueError(f"{field} must be between 0 and 1")
return value
def retry_storm_windows(points, attempt_ratio_threshold=1.8, dependency_error_threshold=0.05):
attempt_ratio_threshold = _finite_non_negative(attempt_ratio_threshold, "attempt_ratio_threshold")
dependency_error_threshold = _finite_rate(dependency_error_threshold, "dependency_error_threshold")
if attempt_ratio_threshold <= 0:
raise ValueError("thresholds must be non-negative and attempt ratio must be positive")
alerts = []
for index, point in enumerate(points):
field = f"points[{index}]"
point = _mapping(point, field)
user_turns = _finite_non_negative(_required(point, "user_turns", field), f"{field}.user_turns")
service_attempts = _finite_non_negative(_required(point, "service_attempts", field), f"{field}.service_attempts")
dependency_error_rate = _finite_rate(_required(point, "dependency_error_rate", field), f"{field}.dependency_error_rate")
user_turn_denominator = max(user_turns, 1)
attempt_ratio = service_attempts / user_turn_denominator
if attempt_ratio >= attempt_ratio_threshold and dependency_error_rate >= dependency_error_threshold:
alerts.append({
"minute": _required(point, "minute", field),
"attempt_ratio": round(attempt_ratio, 2),
"dependency_error_rate": dependency_error_rate,
"action": "cap_retries_and_enable_fallback",
})
return alertsSummarize a step-load test and find the first concurrency level where the system violates its latency or error budget.
Invariant: the supported capacity is the last passing step before the first failing step. In production, repeat or soak that boundary before publishing a capacity number so a single noisy or malformed point does not define the launch limit.
from collections.abc import Mapping
from math import isfinite
def _mapping(value, field):
if not isinstance(value, Mapping):
raise ValueError(f"{field} must be a mapping")
return value
def _required(mapping, key, field):
if key not in mapping:
raise ValueError(f"{field}.{key} is required")
return mapping[key]
def _finite_non_negative(value, field):
if isinstance(value, bool) or not isinstance(value, (int, float)) or not isfinite(value) or value < 0:
raise ValueError(f"{field} must be a finite non-negative number")
return value
def _finite_rate(value, field):
value = _finite_non_negative(value, field)
if value > 1:
raise ValueError(f"{field} must be between 0 and 1")
return value
def first_capacity_break(steps, p95_budget_ms, error_budget):
p95_budget_ms = _finite_non_negative(p95_budget_ms, "p95_budget_ms")
error_budget = _finite_rate(error_budget, "error_budget")
last_passing = None
for index, step in enumerate(steps):
field = f"steps[{index}]"
step = _mapping(step, field)
concurrency = _finite_non_negative(_required(step, "concurrency", field), f"{field}.concurrency")
p95_ms = _finite_non_negative(_required(step, "p95_ms", field), f"{field}.p95_ms")
error_rate = _finite_rate(_required(step, "error_rate", field), f"{field}.error_rate")
passes = p95_ms <= p95_budget_ms and error_rate <= error_budget
if passes:
last_passing = concurrency
continue
return {
"max_supported_concurrency": last_passing,
"first_failing_concurrency": concurrency,
"p95_ms": p95_ms,
"error_rate": error_rate,
}
return {"max_supported_concurrency": last_passing, "first_failing_concurrency": None}Interview And Exam Prompts
You have two weeks before launch. Design the load test, quality gates, rollback triggers, and dashboards for a speech-to-speech support agent serving enterprise tenants.
Cover traffic modeling, tenant skew, approved synthetic fixtures, ASR partial latency, LLM first token, retrieval latency, TTS first audio byte, barge-in, task success, groundedness, safety, cost per success, and rollback. Include real-time versus batch isolation, priority queues, fallback model routes, per-tenant dashboards, and a go/no-go meeting with explicit launch gates.
Give a concise strong answer, then map both to ASR, TTS, and spoken RAG.
Load testing asks whether the system meets promises under expected and peak traffic. Chaos testing asks whether it degrades safely when parts fail. For ASR, load tests stress concurrency and chunking; chaos tests remove GPU capacity or slow VAD. For TTS, load tests stress voice pools and cache; chaos tests bypass cache or cold-start regions. For spoken RAG, load tests stress retrieval fanout; chaos tests add stale indexes, slow dependencies, and ACL uncertainty.