Speech Serving Scaling And Reliability Exam

Start with a streaming gateway, per-call session state, VAD, ASR stream workers, retrieval and policy services, LLM serving, TTS streaming, and playback telemetry. Isolate real-time traffic from batch jobs, reserve warm model pools, and use priority queues for interactive turns. Track concurrent streams, audio seconds per second, partial latency, finalization latency, token throughput, TTS first audio byte, error budget burn, and cost per successful turn. Roll out by language, region, device class, and tenant.

Route by observable aggregate-safe signals such as language, device, expected noise class, tenant tier, real-time requirement, and confidence from early chunks. Use the large model for hard slices, high-value calls, low confidence, or escalation. Gate the policy on WER/CER slices, first partial latency, abandonment, downstream task success, and cost per successful minute. Failure modes include biased routing, stale noise classifiers, retry loops, and silently routing rare accents to the wrong model.

Give every artifact a version, owner, data lineage record, eval report, and rollback target. CI validates schemas, model cards, safety checks, privacy constraints, and reproducible eval inputs. Pre-prod gates cover noisy ASR queries, retrieval freshness, grounded answers, tool-call safety, TTS latency, and end-to-end spoken-task success. Canary gates compare slice metrics against the previous bundle, not only global averages. Rollback can pin one artifact, restore a bundle, disable a feature, or route to a safer fallback.

Split by stage latency, model version, route, language, region, request length, prompt tokens, retrieval document count, TTS voice, and GPU queue depth. Averages can stay flat when one slice has a bad tail. Check continuous batching, KV-cache eviction, prompt growth, cache misses, retry storms, and slow TTS warmup. Roll back or freeze the canary if p99 burn persists for the real-time SLO window, even without a 5xx spike.

Compare slice-level WER, entity error rate, endpointing errors, partial churn, and downstream task success before and after the refresh. Audit labeling mix, augmentation, sample weights, language detection, and test-set leakage. Mitigate by routing the slice to the last good model or raising confidence thresholds. Add active learning coverage checks, label audit gates, and drift monitors for the affected slice.

Inspect tokens per turn, retrieved chunks, prompt template size, retries, context carryover, cache hit rate, model mix, batch occupancy, speculative decoding acceptance, and fallback frequency. Stabilize with token caps, retrieval caps, priority classes, fallback routing, and retry budgets. Prevention is a release gate on cost per successful turn and a dashboard that ties spend to quality, latency, and model version.

Invariant: a sweep-line count equals active streams after applying every event at a timestamp. Process end events before start events when an interval ending at t does not overlap an interval starting at t. Test empty input, touching intervals, nested intervals, identical starts, and one long call.

def peak_concurrent_streams(intervals):
    events = []
    for start, end in intervals:
        events.append((start, 1))
        events.append((end, -1))

    active = 0
    peak = 0
    for _, delta in sorted(events, key=lambda item: (item[0], item[1])):
        active += delta
        peak = max(peak, active)
    return peak

Invariant: promotion is safe only if every protected slice stays within agreed thresholds. Test missing slices, improved quality with worse latency, low-volume slices, and zero old cost. A strong answer describes minimum sample rules instead of trusting tiny samples.

def rollback_blockers(old, new, min_calls=200):
    blockers = []
    for slice_name, current in new.items():
        baseline = old.get(slice_name)
        if baseline is None or current["calls"] < min_calls:
            continue

        wer_delta = current["wer"] - baseline["wer"]
        p95_delta = current["p95_ms"] - baseline["p95_ms"]
        cost_ratio = current["cost_per_min"] / max(baseline["cost_per_min"], 0.01)

        reasons = []
        if wer_delta > 0.015:
            reasons.append("quality")
        if p95_delta > 250:
            reasons.append("latency")
        if cost_ratio > 1.2:
            reasons.append("cost")
        if reasons:
            blockers.append({"slice": slice_name, "reasons": reasons})
    return blockers

This is the same window invariant as Blind 75 substring problems: maintain exactly the aggregate state for the current window, update it when the right edge enters, and remove the left edge when it leaves. Test all-good traffic, all-bad traffic, zero traffic, and a single bad spike.

def burn_windows(points, window_size=5, budget_rate=0.01):
    bad = 0
    total = 0
    alerts = []

    for right, point in enumerate(points):
        bad += point["bad"]
        total += point["total"]

        if right >= window_size:
            left = points[right - window_size]
            bad -= left["bad"]
            total -= left["total"]

        if right + 1 >= window_size and total > 0:
            rate = bad / total
            if rate > budget_rate:
                alerts.append({
                    "ending_minute": point["minute"],
                    "error_rate": rate,
                    "window_size": window_size,
                })
    return alerts