Speech MLOps, Serving, And Incident Response

Check endpointing, chunk size, partial stabilization rules, decoder beam settings, language-model rescoring, VAD thresholds, network jitter, and whether the UI is rendering interim hypotheses as committed text. Add a metric for partial churn and compare old and new traces on the same audio fixtures.

Measure text normalization, acoustic model time, vocoder time, streaming chunk release, cold starts, and device placement. Mitigate with rollback, canary pause, warm pools, smaller voice, lower vocoder quality for long responses, sentence-level streaming, cache common prompts, or split traffic by latency-sensitive use case. Do not hide latency regressions behind offline MOS.

Compare model version, average audio duration, batch fill rate, padding waste, beam size, max tokens, quantization, cache reuse, autoscaler target, replica count, retries, and failed requests. Check whether a fallback path silently moved from CPU to GPU or whether canary plus shadow traffic doubled inference.

Start with SLOs: first partial transcript, first audio byte, turn completion, task success, cost per minute, and privacy retention. Launch behind flags with synthetic canaries, internal dogfood, small cohort canary, and rollback to the prior assistant. Monitor ASR WER slices, entity error, LLM groundedness, TTS first audio byte, queue time, GPU memory, retry rate, cancellation, and user correction rate. Keep raw audio out of logs by default, store redacted traces, and route high-risk or high-cost sessions to a fallback path.

Compare aggregate acoustic features before and after the firmware: sample rate, clipping, silence ratio, loudness, SNR proxy, channel count, duration, VAD segments, language probabilities, confidence, and correction categories. Reproduce with public fixtures recorded through the same device path, synthetic noisy clips, and opted-in samples if policy allows. Mitigate with input normalization, firmware-specific routing, VAD threshold changes, or rollback of the firmware path. Update eval fixtures so the failure cannot return unnoticed.

Do not decide from global WER alone. Compare slice metrics, entity error, downstream task success, latency, cost, approved operational context, and recovery behavior. A likely strong answer is a cascade: route clean, short, high-confidence utterances to the smaller model and route noisy, long, low-confidence, or domain-heavy utterances to the larger model. Canary the router, cap blast radius, monitor routing mistakes, and keep a one-switch rollback to the baseline.

Require unit tests for feature shape, sample-rate handling, normalization, streaming chunk boundaries, and model wrapper schema. Require repeatable offline eval on public, synthetic, or approved retained fixtures, slice regression checks, latency and memory benchmarks, artifact versioning, dependency lockfile, model card, rollback plan, and privacy review. The deployment should stage through offline eval, policy-approved shadow traffic, canary, and gradual rollout with automatic halt criteria.

A strong gate treats approved launch-critical slices as first-class release criteria. Merge should be blocked or paused when a critical slice exceeds its regression budget, even if the aggregate metric improves. Include invalid-budget, missing-slice, and missing-metric handling, plus range checks for WER values, so eval coverage and metric quality cannot silently disappear.

import math
from collections.abc import Mapping


def valid_rate(value):
    return (
        not isinstance(value, bool)
        and isinstance(value, (int, float))
        and math.isfinite(value)
        and 0 <= value <= 1
    )


def asr_slice_gate(baseline, candidate, budgets):
    if not all(isinstance(group, Mapping) for group in (baseline, candidate, budgets)):
        return "pause_for_eval", [("pause", "all", "metric inputs must be maps")]

    findings = []
    for slice_name, budget in budgets.items():
        if not isinstance(budget, Mapping):
            findings.append(("pause", slice_name, "invalid budget entry"))
            continue
        max_delta = budget.get("max_wer_delta")
        if not valid_rate(max_delta):
            findings.append(("pause", slice_name, "invalid regression budget"))
            continue
        if slice_name not in baseline or slice_name not in candidate:
            findings.append(("pause", slice_name, "missing eval slice"))
            continue
        old_metrics = baseline[slice_name]
        new_metrics = candidate[slice_name]
        if not isinstance(old_metrics, Mapping) or not isinstance(new_metrics, Mapping):
            findings.append(("pause", slice_name, "invalid metric entry"))
            continue
        if "wer" not in old_metrics or "wer" not in new_metrics:
            findings.append(("pause", slice_name, "missing WER metric"))
            continue
        old_wer = old_metrics["wer"]
        new_wer = new_metrics["wer"]
        if not valid_rate(old_wer) or not valid_rate(new_wer):
            findings.append(("pause", slice_name, "WER outside 0..1 range"))
            continue
        delta = new_wer - old_wer
        if delta > max_delta:
            findings.append(("block", slice_name, round(delta, 4)))

    if any(item[0] == "block" for item in findings):
        return "block_merge", findings
    if findings:
        return "pause_for_eval", findings
    return "merge_allowed", []

Include model version, feature flag name, owner, launch cohort, dashboard links, SLO thresholds, known risks, exact rollback command or UI path, data compatibility notes, cache behavior, retention limits, verification query, customer-support message, and criteria for re-enabling. The packet should avoid private transcripts and raw audio while still giving enough aggregate metrics and redacted traces to make a decision.

Break the budget into capture/VAD, ASR partial, retrieval, first LLM tokens, TTS first audio byte, network, and client playback. Use streaming overlap instead of serial totals where possible: ASR partials can trigger early retrieval, the LLM can emit a short first clause, and TTS can stream before the full answer exists. State which stages need warm pools, which can batch, and which should bypass expensive models for low-risk turns.

First hour: pause or roll back the canary for the affected approved aggregate slice, compare request routing, language ID, contextual biasing, text normalization, and entity errors against the baseline. Next day: build a targeted eval slice, inspect redacted or approved examples, check label quality, and test routing to the prior model for mixed language turns. Prevention: add code-switching gates, dashboard slices, launch alerts, and a release note requiring language-mix analysis for ASR changes.