ML System Design For Audio-Text Products

Separate real-time streaming from offline correction jobs. Use VAD, normalization, streaming ASR, domain contextual biasing, entity post-processing, and privacy-safe aggregate telemetry. Gate releases by WER, entity error rate, partial churn, first partial latency, finalization delay, tenant-level SLOs, and correction rate. Roll back by model version and feature flag, not by redeploying the whole service.

A cascaded ASR-LLM/RAG-TTS system is often easier to debug and control than a direct speech-to-speech model, while direct models may reduce latency or preserve prosody when they have strong evals and guardrails. Track ASR confidence, retrieval grounding, answer safety, TTS first audio byte, turn latency, barge-in, and handoff rate. Use session state with explicit TTLs, policy-document versioning, retrieval evals from spoken queries, and human escalation when grounding, identity, or intent confidence is weak.

Split interactive and batch capacity, enforce quotas, provide model contracts, support continuous batching where it helps, and expose per-model cost and latency dashboards. Use priority queues, request deadlines, admission control, canary routing, model pools, warmup, autoscaling, and fallback tiers. Cost savings should be gated by slice quality, not only aggregate throughput.

Collect opt-in examples or derived features under retention rules, label with clear rubrics, measure annotator agreement, and build eval slices before training changes. Store dataset lineage, model version, prompt/config version, and consent state. Use synthetic and public fixtures for CI, reserve private data for governed evals, and monitor drift through aggregate features and correction signals.

Compare old and new voice by first audio byte, chunk cadence, text normalization, sentence segmentation, vocoder time, client playback, interruption rate, abandonment, and approved aggregate language slices. Mitigate with canary rollback, fallback voice, shorter first segment, warm pools, or lower-cost vocoding for noncritical turns. Prevent with conversational evals, not only offline preference MOS.

Invariant: every metric is compared in the direction that matters for the product. Test quality regression with latency win, latency regression with quality win, missing metrics, invalid budget definitions, and exactly-on-budget changes.

def canary_decision(baseline, canary, budgets):
    actions = []
    for name, budget in budgets.items():
        required_fields = {"direction", "allowed_delta"}
        missing_fields = required_fields - budget.keys()
        if missing_fields:
            missing = ", ".join(sorted(missing_fields))
            raise ValueError(f"budget for {name} missing: {missing}")

        if name not in baseline or name not in canary:
            actions.append(("pause", name, "missing metric"))
            continue

        direction = budget["direction"]
        allowed = budget["allowed_delta"]
        if allowed < 0:
            raise ValueError("allowed_delta must be non-negative")

        delta = canary[name] - baseline[name]
        tolerance = 1e-12
        if direction == "lower_is_better" and delta - allowed > tolerance:
            actions.append(("rollback", name, delta))
        elif direction == "higher_is_better" and -delta - allowed > tolerance:
            actions.append(("rollback", name, delta))
        elif direction not in {"lower_is_better", "higher_is_better"}:
            raise ValueError(f"unknown direction for {name}: {direction}")

    if any(action[0] == "rollback" for action in actions):
        return "rollback", actions
    if actions:
        return "pause", actions
    return "continue", []

Invariant: real-time streams produce audio work every wall-clock second, reduced only by silence skipping or routing. Test zero streams, no VAD savings, invalid utilization, invalid negative inputs, nonpositive replica capacity, and very large peaks.

import math


def estimate_replicas(concurrent_streams, gpu_seconds_per_audio_second,
                      replica_capacity=1.0, target_utilization=0.7,
                      vad_savings=0.0):
    if not 0 < target_utilization <= 1:
        raise ValueError("target_utilization must be in (0, 1]")
    if not 0 <= vad_savings < 1:
        raise ValueError("vad_savings must be in [0, 1)")
    if replica_capacity <= 0:
        raise ValueError("replica_capacity must be positive")
    if concurrent_streams < 0:
        raise ValueError("concurrent_streams must be non-negative")
    if gpu_seconds_per_audio_second < 0:
        raise ValueError("gpu_seconds_per_audio_second must be non-negative")

    effective_audio_seconds = concurrent_streams * (1 - vad_savings)
    gpu_seconds_needed = effective_audio_seconds * gpu_seconds_per_audio_second
    usable_capacity = replica_capacity * target_utilization
    return math.ceil(gpu_seconds_needed / usable_capacity)

Roll back when user harm, privacy risk, SLO breach, critical slice regression, data corruption, runaway cost, or unrecoverable queue growth exceeds the launch budget. Debug after stabilizing unless the rollback itself would create larger risk.

Spoken queries include ASR substitutions, punctuation loss, entity errors, disfluencies, approved aggregate pronunciation-variation slices, language switches, endpointing mistakes, and rewritten partials. Evaluate clean text, final ASR, noisy ASR, and streaming partial paths separately.