ML System Design For Audio-Text Products

Separate real-time streaming from offline correction jobs. Use VAD, normalization, streaming ASR, domain contextual biasing, entity post-processing, and privacy-safe telemetry. Gate releases by WER, entity error rate, partial churn, first partial latency, finalization delay, tenant-level SLOs, and correction rate. Roll back by model version and feature flag, not by redeploying the whole service.

A cascaded ASR-LLM/RAG-TTS system is easier to debug and control than a direct speech-to-speech model. Track ASR confidence, retrieval grounding, answer safety, TTS first audio byte, turn latency, barge-in, and handoff rate. Use session state with explicit TTLs, policy-document versioning, retrieval evals from spoken queries, and human escalation when grounding, identity, or intent confidence is weak.

Split interactive and batch capacity, enforce quotas, provide model contracts, support continuous batching where it helps, and expose per-model cost and latency dashboards. Use priority queues, request deadlines, admission control, canary routing, model pools, warmup, autoscaling, and fallback tiers. Cost savings should be gated by slice quality, not only aggregate throughput.

Collect opt-in examples or derived features under retention rules, label with clear rubrics, measure annotator agreement, and build eval slices before training changes. Store dataset lineage, model version, prompt/config version, and consent state. Use synthetic and public fixtures for CI, reserve private data for governed evals, and monitor drift through aggregate features and correction signals.

Compare old and new voice by first audio byte, chunk cadence, text normalization, sentence segmentation, vocoder time, client playback, interruption rate, abandonment, and long-tail languages. Mitigate with canary rollback, fallback voice, shorter first segment, warm pools, or lower-cost vocoding for noncritical turns. Prevent with conversational evals, not only offline preference MOS.

Invariant: every metric is compared in the direction that matters for the product. Test quality regression with latency win, latency regression with quality win, missing metrics, and exactly-on-budget changes.

def canary_decision(baseline, canary, budgets):
    actions = []
    for name, budget in budgets.items():
        if name not in baseline or name not in canary:
            actions.append(("pause", name, "missing metric"))
            continue

        direction = budget["direction"]
        allowed = budget["allowed_delta"]
        delta = canary[name] - baseline[name]
        if direction == "lower_is_better" and delta > allowed:
            actions.append(("rollback", name, delta))
        elif direction == "higher_is_better" and -delta > allowed:
            actions.append(("rollback", name, delta))

    if any(action[0] == "rollback" for action in actions):
        return "rollback", actions
    if actions:
        return "pause", actions
    return "continue", []

Invariant: real-time streams produce audio work every wall-clock second, reduced only by silence skipping or routing. Test zero streams, no VAD savings, invalid utilization, and very large peaks.

import math


def estimate_replicas(concurrent_streams, gpu_seconds_per_audio_second,
                      target_utilization=0.7, vad_savings=0.0):
    if not 0 < target_utilization <= 1:
        raise ValueError("target_utilization must be in (0, 1]")
    if not 0 <= vad_savings < 1:
        raise ValueError("vad_savings must be in [0, 1)")

    effective_audio_seconds = concurrent_streams * (1 - vad_savings)
    gpu_seconds_needed = effective_audio_seconds * gpu_seconds_per_audio_second
    return math.ceil(gpu_seconds_needed / target_utilization)

Roll back when user harm, privacy risk, SLO breach, critical slice regression, data corruption, runaway cost, or unrecoverable queue growth exceeds the launch budget. Debug after stabilizing unless the rollback itself would create larger risk.

Spoken queries include ASR substitutions, punctuation loss, entity errors, disfluencies, accents, language switches, endpointing mistakes, and rewritten partials. Evaluate clean text, final ASR, noisy ASR, and streaming partial paths separately.