Speech MLOps, Serving, And Incident Response

Check endpointing, chunk size, partial stabilization rules, decoder beam settings, language-model rescoring, VAD thresholds, network jitter, and whether the UI is rendering interim hypotheses as committed text. Add a metric for partial churn and compare old and new traces on the same audio fixtures.

Measure text normalization, acoustic model time, vocoder time, streaming chunk release, cold starts, and device placement. Mitigate with rollback, canary pause, warm pools, smaller voice, lower vocoder quality for long responses, sentence-level streaming, cache common prompts, or split traffic by latency-sensitive use case. Do not hide latency regressions behind offline MOS.

Compare model version, average audio duration, batch fill rate, padding waste, beam size, max tokens, quantization, cache reuse, autoscaler target, replica count, retries, and failed requests. Check whether a fallback path silently moved from CPU to GPU or whether canary plus shadow traffic doubled inference.

Start with SLOs: first partial transcript, first audio byte, turn completion, task success, cost per minute, and privacy retention. Launch behind flags with synthetic canaries, internal dogfood, small cohort canary, and rollback to the prior assistant. Monitor ASR WER slices, entity error, LLM groundedness, TTS first audio byte, queue time, GPU memory, retry rate, cancellation, and user correction rate. Keep raw audio out of logs by default, store redacted traces, and route high-risk or high-cost sessions to a fallback path.

Compare aggregate acoustic features before and after the firmware: sample rate, clipping, silence ratio, loudness, SNR proxy, channel count, duration, VAD segments, language probabilities, confidence, and correction categories. Reproduce with public fixtures recorded through the same device path, synthetic noisy clips, and opted-in samples if policy allows. Mitigate with input normalization, firmware-specific routing, VAD threshold changes, or rollback of the firmware path. Update eval fixtures so the failure cannot return unnoticed.

Do not decide from global WER alone. Compare slice metrics, entity error, downstream task success, latency, cost, user segment, and recovery behavior. A likely strong answer is a cascade: route clean, short, high-confidence utterances to the smaller model and route noisy, long, low-confidence, or domain-heavy utterances to the larger model. Canary the router, cap blast radius, monitor routing mistakes, and keep a one-switch rollback to the baseline.

Require unit tests for feature shape, sample-rate handling, normalization, streaming chunk boundaries, and model wrapper schema. Require deterministic eval on public or approved fixtures, slice regression checks, latency and memory benchmarks, artifact versioning, dependency lockfile, model card, rollback plan, and privacy review. The deployment should stage through offline eval, shadow traffic, canary, and gradual rollout with automatic halt criteria.

A strong gate treats protected slices as first-class release criteria. Merge should be blocked or paused when a critical slice exceeds its regression budget, even if the aggregate metric improves. Include missing-slice handling so eval coverage cannot silently disappear.

def asr_slice_gate(baseline, candidate, budgets):
    findings = []
    for slice_name, budget in budgets.items():
        if slice_name not in baseline or slice_name not in candidate:
            findings.append(("pause", slice_name, "missing eval slice"))
            continue
        old_wer = baseline[slice_name]["wer"]
        new_wer = candidate[slice_name]["wer"]
        delta = new_wer - old_wer
        if delta > budget["max_wer_delta"]:
            findings.append(("block", slice_name, round(delta, 4)))

    if any(item[0] == "block" for item in findings):
        return "block_merge", findings
    if findings:
        return "pause_for_eval", findings
    return "merge_allowed", []

Include model version, feature flag name, owner, launch cohort, dashboard links, SLO thresholds, known risks, exact rollback command or UI path, data compatibility notes, cache behavior, verification query, customer-support message, and criteria for re-enabling. The packet should avoid private transcripts while still giving enough metrics and traces to make a decision.

Break the budget into capture/VAD, ASR partial, retrieval, first LLM tokens, TTS first audio byte, network, and client playback. Use streaming overlap instead of serial totals where possible: ASR partials can trigger early retrieval, the LLM can emit a short first clause, and TTS can stream before the full answer exists. State which stages need warm pools, which can batch, and which should bypass expensive models for low-risk turns.

First hour: pause or roll back the canary for the affected slice, compare request routing, language ID, contextual biasing, text normalization, and entity errors against the baseline. Next day: build a targeted eval slice, inspect redacted or approved examples, check label quality, and test routing to the prior model for mixed language turns. Prevention: add code-switching gates, dashboard slices, launch alerts, and a release note requiring language-mix analysis for ASR changes.