Speech Serving Observability And SLOs

Invariant: burn rate compares observed bad fraction to the allowed bad fraction. Test zero requests, no errors, exactly-on-budget errors, invalid negative counts, bad counts above total, invalid budget fractions, and a window that burns several times faster than budget. The function below scores one window; a real pager should combine short and long windows before waking an owner.

def burn_rate(total_requests, bad_requests, allowed_bad_fraction):
    if total_requests < 0 or bad_requests < 0:
        raise ValueError("request counts must be non-negative")
    if bad_requests > total_requests:
        raise ValueError("bad_requests cannot exceed total_requests")
    if allowed_bad_fraction <= 0 or allowed_bad_fraction > 1:
        raise ValueError("allowed_bad_fraction must be in (0, 1]")
    if total_requests == 0:
        return 0.0

    observed = bad_requests / total_requests
    return observed / allowed_bad_fraction

Invariant: report slices that appear or disappear, and compare a slice only when both baseline and current values exist. Treat a zero baseline as a special case so the code does not hide a new population. Test missing slices, zero baseline, invalid negative thresholds, small changes, and large increases in silence or clipping.

def drifted_slices(baseline, current, threshold):
    if threshold < 0:
        raise ValueError("threshold must be non-negative")

    drifted = []
    for name in sorted(set(baseline) | set(current)):
        if name not in baseline:
            now = current[name]
            drifted.append((name, "new_slice", now))
            continue
        if name not in current:
            drifted.append((name, "missing_current", baseline[name]))
            continue

        now = current[name]
        before = baseline[name]
        if before == 0:
            if now != 0:
                drifted.append((name, "zero_baseline", now))
            continue

        relative = abs(now - before) / abs(before)
        if relative > threshold:
            drifted.append((name, "relative_change", relative))
    return drifted

Invariant: critical metrics override noncritical wins. Test a latency win with entity error regression, missing critical metrics, an invalid direction string, exactly-on-budget deltas, and a noncritical cost regression that should pause instead of roll back.

def rollout_recommendation(metrics, budgets):
    findings = []
    for name, budget in budgets.items():
        if name not in metrics:
            findings.append(("pause", name, "missing"))
            continue

        value = metrics[name]
        limit = budget["limit"]
        direction = budget["direction"]
        if direction not in {"lower_is_better", "higher_is_better"}:
            raise ValueError(f"unknown direction for {name}: {direction}")

        critical = budget.get("critical", False)
        failed = (
            direction == "lower_is_better" and value > limit
        ) or (
            direction == "higher_is_better" and value < limit
        )
        if failed:
            action = "rollback" if critical else "pause"
            findings.append((action, name, value))

    if any(item[0] == "rollback" for item in findings):
        return "rollback", findings
    if findings:
        return "pause", findings
    return "continue", []

Do not hide behind the aggregate win. Pause or exclude the affected cohort, verify sample size and label quality, inspect audio feature drift, compare entity error and correction rate, and define a slice gate using approved aggregate or consented tags before resuming. A strong answer protects harmed cohorts while preserving the ability to ship where the model is truly better.

Check model artifact size, precision, batch shape, request mix, VAD savings, retries, shadow traffic, autoscaler target, warm pool size, GPU utilization, cache behavior, and route weights. Mitigate with quota, route rollback, batch tuning, disabling shadow traffic, or reverting precision/config changes while preserving quality.

Include timeline, detection gap, user impact by approved aggregate or consented slice, root cause, why tests missed quiet speakers, rollback timeline, privacy-safe evidence, immediate remediation, new synthetic fixtures, release gate updates, owners, deadlines, and a follow-up check that proves the prevention work actually shipped.