Model Serving On-Call Runbook For Speech AI

Slice by model version, device, codec, language, VAD state, chunk size, region, and connection type. Check VAD false silence, audio chunk buffering, batching policy, GPU queue depth, and client upload cadence. Mitigate with canary rollback or reduced batch delay on interactive traffic. Add a release gate for first partial latency and partial churn by noisy/mobile slices, not only final WER.

Compare text normalization, sentence segmentation, acoustic model warmup, vocoder time, cache hit rate, region placement, and playback errors. Mitigate by rolling back the voice for interactive flows, using a short first segment, warming pools, or routing long-form synthesis to batch capacity. Prevention: add conversational latency, interruption rate, and abandonment gates alongside MOS/preference.

Look for prompt growth, retrieval returning too many documents, longer conversations, disabled caching, lower batch efficiency, speculative decoding fallback, model mix changes, and retry storms. Stabilize with token budgets, admission control, priority queues, fallback models, and retrieval caps. Add cost-per-success and tokens-per-turn gates to release reviews.

Check index version, document ACLs, embedding model version, chunking, freshness watermark, ASR entity substitutions, reranker logs, and prompt grounding instructions. Mitigate by pinning the tenant to the last good index or forcing human handoff for affected intents. Prevention: add per-tenant retrieval freshness and grounded answer evals with noisy ASR queries before index promotion.

Invariant: overload is a joint signal. High utilization alone can be healthy; queue growth plus tail latency means work is waiting. Test flat queues, rising queues with low latency, high latency with no queue growth, and missing minutes.

def detect_saturation(minutes, queue_growth_threshold=25, p95_threshold_ms=1200):
    alerts = []
    previous_depth = None

    for point in minutes:
        depth = point["queue_depth"]
        p95 = point["p95_ms"]
        if previous_depth is None:
            previous_depth = depth
            continue

        growth = depth - previous_depth
        if growth >= queue_growth_threshold and p95 >= p95_threshold_ms:
            alerts.append({
                "minute": point["minute"],
                "queue_growth": growth,
                "p95_ms": p95,
                "action": "shed_batch_or_add_capacity",
            })
        previous_depth = depth

    return alerts

Common mistakes are dividing by all turns instead of successful turns, ignoring zero-success routes, and comparing cost without quality. A strong answer treats cost as a constraint attached to user value, not as an isolated infrastructure metric.

def rank_cost_per_success(routes):
    ranked = []
    for name, metrics in routes.items():
        successes = metrics["turns"] * metrics["task_success_rate"]
        if successes == 0:
            cost_per_success = float("inf")
        else:
            cost_per_success = metrics["total_cost_usd"] / successes
        ranked.append((cost_per_success, name, metrics["task_success_rate"]))

    return sorted(ranked, reverse=True)

Invariant: a critical-slice regression can outweigh a larger aggregate win. Weight user-harm slices such as emergency intents, medical vocabulary, identity verification, or payment flows higher than generic traffic.

def select_rollback_candidate(version_slices, critical_weights):
    scores = {}
    for version, slices in version_slices.items():
        score = 0.0
        for slice_name, regression in slices.items():
            weight = critical_weights.get(slice_name, 1.0)
            score += max(0.0, regression) * weight
        scores[version] = score

    version, score = max(scores.items(), key=lambda item: item[1])
    return None if score == 0 else version

Interactive traffic has strict latency and deadline requirements, while batch work values throughput and cost efficiency. Shared queues let cheap batch work create tail-latency incidents for live conversations. Isolation enables priority scheduling, admission control, predictable SLOs, and independent scaling policies.

If the LLM path is slow but ASR and account lookup are healthy, the system can degrade to shorter answers, fewer retrieved documents, a smaller model, or human handoff for high-risk intents. Rollback is better when a known release created user harm or quality regression; degradation is better when demand temporarily exceeds capacity and the reduced path still satisfies the product contract.

Use aggregate slice metrics, consented eval sets, synthetic fixtures, hashed identifiers, redacted examples, confusion classes, correction rates, endpointing statistics, and private review tools with access controls. Incident tickets should carry enough evidence to reproduce the failure class without exposing sensitive content.