Speech GPU Serving Capacity Planning

Start from concurrent streams and audio seconds per second, then split by language, model, region, and real-time priority. Measure max streams per replica at p95 first-partial latency with realistic noise and endpointing. Reserve zone-loss headroom, isolate eval and replay traffic, and autoscale on queue age plus active streams rather than GPU utilization alone. Rollout gates should compare WER/CER slices, partial churn, first partial p95/p99, finalization latency, endpointing errors, and cost per successful audio minute.

Inspect tokens per turn, retrieved chunk count, chat-history carryover, cache hit rate, batch occupancy, KV-cache eviction, decode throughput, and route mix. Longer prompts reduce effective batch size and raise prefill latency before TTS can start. Mitigate with retrieval caps, prompt compaction, session summarization, context budgets by tier, speculative decoding checks, and rollback to the previous prompt bundle if first-audio SLO burn continues.

Launch by voice, language, tenant, and region. Keep warm pools for top voices and lazy-load low-volume voices behind a clear latency budget. Track first audio byte, streaming underruns, voice cache hit rate, memory fragmentation, fallback voice usage, MOS proxies, and abuse signals. Protect real-time traffic with priority queues and fall back to a smaller or neutral voice if cold-start queues exceed the SLO window.

Scale ASR on active streams, chunk queue age, first partial p95, and CPU feature extraction saturation. Scale LLM serving on prefill queue age, active tokens, KV-cache pressure, decode tokens per second, and time to first token. Scale TTS on first audio byte, active synthesis streams, warm voice cache misses, and underruns. Load shedding should degrade in order: reduce retrieval depth, shorten context, route to smaller models, delay non-real-time jobs, disable premium effects, and finally reject low-priority traffic with clear retry behavior.

Split by stage, traffic class, region, language, voice, model bundle, queue age, and retry count. Check whether batch or replay jobs leaked into the real-time pool, whether one dependency is causing backpressure, and whether autoscaling is waiting on cold starts. Freeze rollouts, drain non-real-time jobs, route affected slices to warm fallback pools, and roll back the latest bundle if a version-linked tail regression persists.

Compare cost per successful turn, model mix, batch occupancy, speculative acceptance rate, prompt tokens, cache reuse, fallback rate, and overprovisioned warm pools. Better latency may come from wasteful headroom or smaller batches. Decide whether the product needs that latency gain, then set tier-specific SLOs, tighter warm pool targets, context budgets, and a release gate on cost per successful task rather than raw latency alone.

Invariant: the running sum always equals the current window. Common mistakes are off-by-one return times, integer division, and forgetting that alerting usually triggers at the end of the window. Test empty input, k larger than input, threshold exactly equal, first window breach, later breach, and no breach.

def first_capacity_alert(queue_age_ms, k, threshold_ms):
    if k <= 0:
        raise ValueError("k must be positive")
    if len(queue_age_ms) < k:
        return None

    window = sum(queue_age_ms[:k])
    if window / k > threshold_ms:
        return k - 1

    for right in range(k, len(queue_age_ms)):
        window += queue_age_ms[right] - queue_age_ms[right - k]
        if window / k > threshold_ms:
            return right
    return None

This is a sorting and greedy policy question, not optimal knapsack unless the interviewer asks for value maximization. State the product policy first. Test zero capacity, ties, requests larger than capacity, equal priorities with different costs, and stable identifiers for admitted requests.

def admit_requests(requests, capacity):
    ordered = sorted(
        requests,
        key=lambda r: (-r["priority"], r["estimated_cost"], r["id"]),
    )
    admitted = []
    used = 0
    for request in ordered:
        cost = request["estimated_cost"]
        if used + cost <= capacity:
            admitted.append(request["id"])
            used += cost
    return admitted

Maintain rolling requests and retries. The invariant is that both sums represent exactly the current window. Avoid dividing by zero, alerting on tiny samples, or looking only at retry count without traffic volume.

def retry_storm_windows(records, k, retry_threshold, min_requests):
    if k <= 0:
        raise ValueError("k must be positive")
    alerts = []
    req_sum = 0
    retry_sum = 0

    for i, row in enumerate(records):
        req_sum += row["requests"]
        retry_sum += row["retries"]
        if i >= k:
            old = records[i - k]
            req_sum -= old["requests"]
            retry_sum -= old["retries"]
        if i >= k - 1 and req_sum >= min_requests:
            if req_sum and retry_sum / req_sum > retry_threshold:
                alerts.append(row["minute"])
    return alerts