Speech Serving Scaling and Reliability Exam

Start with a streaming gateway, per-call session state, VAD, ASR stream workers, retrieval and policy services, LLM serving, TTS streaming, and playback telemetry. Isolate real-time traffic from batch jobs, reserve warm model pools, and use priority queues for interactive turns. Track concurrent streams, audio seconds per second, partial latency, finalization latency, token throughput, TTS first audio byte, error budget burn, and cost per successful turn. Roll out by language, region, device class, and tenant.

Route by observable aggregate-safe signals such as language, device, expected noise class, tenant tier, real-time requirement, and confidence from early chunks. Use the large model for hard slices, high-value calls, low confidence, or escalation. Gate the policy on WER/CER slices, first partial latency, abandonment, downstream task success, and cost per successful minute. Failure modes include biased routing, stale noise classifiers, retry loops, and silently routing approved aggregate or consented pronunciation variation slices to the wrong model.

Give every artifact a version, owner, data lineage record, eval report, and rollback target. CI validates schemas, model cards, safety checks, privacy constraints, and reproducible eval inputs. Pre-prod gates cover noisy ASR queries, retrieval freshness, grounded answers, tool-call safety, TTS latency, and end-to-end spoken-task success. Canary gates compare slice metrics against the previous bundle, not only global averages. Rollback can pin one artifact, restore a bundle, disable a feature, or route to a safer fallback.

Split by stage latency, model version, route, language, region, request length, prompt tokens, retrieval document count, TTS voice, and GPU queue depth. Averages can stay flat when one slice has a bad tail. Check continuous batching, KV-cache eviction, prompt growth, cache misses, retry storms, and slow TTS warmup. Roll back or freeze the canary if p99 burn persists for the real-time SLO window, even without a 5xx spike.

Compare privacy-approved slice-level WER, entity error rate, endpointing errors, partial churn, and downstream task success before and after the refresh. Audit labeling mix, augmentation, sample weights, language detection, consent or aggregation policy, and test-set leakage. Mitigate by routing the slice to the last good model or raising confidence thresholds. Add active learning coverage checks, label audit gates, and drift monitors for the affected approved aggregate or consented slice.

Inspect tokens per turn, retrieved chunks, prompt template size, retries, context carryover, cache hit rate, model mix, batch occupancy, speculative decoding acceptance, and fallback frequency. Stabilize with token caps, retrieval caps, priority classes, fallback routing, and retry budgets. Prevention is a release gate on cost per successful turn and a dashboard that ties spend to quality, latency, and model version.

Invariant: a sweep-line count equals active streams after applying every event at a timestamp. Treat intervals as half-open [start_ms, end_ms) call spans and validate that every call has positive duration before the sweep. Process end events before start events when an interval ending at t does not overlap an interval starting at t. Test empty input, touching intervals, nested intervals, identical starts, zero-length intervals, inverted intervals, and one long call.

def peak_concurrent_streams(intervals):
    events = []
    for start, end in intervals:
        if end <= start:
            raise ValueError("end_ms must be greater than start_ms")
        events.append((start, 1))
        events.append((end, -1))

    active = 0
    peak = 0
    for _, delta in sorted(events, key=lambda item: (item[0], item[1])):
        active += delta
        peak = max(peak, active)
    return peak

Invariant: promotion is safe only if every launch-critical slice stays within agreed thresholds. Test missing slices, improved quality with worse latency, low-volume old or new slices, and zero old cost. A strong answer describes minimum sample rules instead of trusting tiny samples.

def rollback_blockers(old, new, min_calls=200):
    if min_calls <= 0:
        raise ValueError("min_calls must be positive")

    blockers = []
    for slice_name, current in new.items():
        baseline = old.get(slice_name)
        if baseline is None:
            continue

        if current["calls"] < 0 or baseline["calls"] < 0:
            raise ValueError("calls must be non-negative")
        if current["wer"] < 0 or baseline["wer"] < 0:
            raise ValueError("wer must be non-negative")
        if current["p95_ms"] < 0 or baseline["p95_ms"] < 0:
            raise ValueError("p95_ms must be non-negative")
        if current["cost_per_min"] < 0 or baseline["cost_per_min"] < 0:
            raise ValueError("cost_per_min must be non-negative")

        if current["calls"] < min_calls or baseline["calls"] < min_calls:
            continue

        wer_delta = current["wer"] - baseline["wer"]
        p95_delta = current["p95_ms"] - baseline["p95_ms"]
        cost_ratio = current["cost_per_min"] / max(baseline["cost_per_min"], 0.01)

        reasons = []
        if wer_delta > 0.015:
            reasons.append("quality")
        if p95_delta > 250:
            reasons.append("latency")
        if cost_ratio > 1.2:
            reasons.append("cost")
        if reasons:
            blockers.append({"slice": slice_name, "reasons": reasons})
    return blockers

Maintain exactly the aggregate state for the current SLO window, update it when the right edge enters, and remove the left edge when it leaves. Reject non-positive window sizes, negative request counts, and bad counts above total counts before computing burn. Test all-good traffic, all-bad traffic, zero traffic, invalid telemetry, and a single bad spike.

def burn_windows(points, window_size=5, budget_rate=0.01):
    if window_size <= 0:
        raise ValueError("window_size must be positive")
    if budget_rate < 0:
        raise ValueError("budget_rate must be non-negative")

    bad = 0
    total = 0
    alerts = []

    for right, point in enumerate(points):
        if point["bad"] < 0 or point["total"] < 0:
            raise ValueError("request counts must be non-negative")
        if point["bad"] > point["total"]:
            raise ValueError("bad requests cannot exceed total requests")

        bad += point["bad"]
        total += point["total"]

        if right >= window_size:
            left = points[right - window_size]
            bad -= left["bad"]
            total -= left["total"]

        if right + 1 >= window_size and total > 0:
            rate = bad / total
            if rate > budget_rate:
                alerts.append({
                    "ending_minute": point["minute"],
                    "error_rate": rate,
                    "window_size": window_size,
                })
    return alerts