Speech Data And Evaluation Operations

Require aggregate and slice WER, entity error, partial churn, first partial latency, finalization delay, correction rate, timeout rate, retry rate, cost per audio minute, privacy review, rollback flag, and owner sign-off. Set explicit budgets for approved aggregate or consented language and pronunciation-variation slices, noisy mobile audio, rare names, code-switching, and long dictation, and bucket sparse sensitive slices before reporting.

Compare preference lift against latency, interruption rate, completion rate, device buffer behavior, warm-pool cost, and fallback quality. Consider routing short conversational answers to the faster path while using the preferred voice for longer or less latency-sensitive responses.

Pair clean text queries with ASR hypotheses and noisy hypotheses. Measure retrieval recall, entity miss rate, answer groundedness, refusal correctness, citation quality, latency, and spoken response usefulness. Spoken-query regressions often come from ASR substitutions interacting with chunking or filters.

Treat correction rate as product-coupled, not pure model quality. Compare edit affordances, abandoned sessions, manual retyping, thumbs-down feedback, support tickets, final transcript error audits, and cohort-level usage. Add a stable offline eval and a UI-independent human review sample before trusting the metric.

Ask about data rights, retention, latency, streaming partials, domain vocabulary, custom biasing, outages, observability, rollback, cost at peak, privacy review, security boundaries, support SLAs, and slice regressions. Offline WER is only one input to a production replacement decision.

Invariant: each flagged row has a slice present in both metric maps, both metric values are finite, the slice has enough examples for a release gate, new - old is greater than the non-negative regression budget, and missing, newly added, or underpowered slices are reported separately instead of being silently treated as zero.

from math import isfinite
from numbers import Integral, Real


def _finite_number(value, name):
    if isinstance(value, bool) or not isinstance(value, Real) or not isfinite(value):
        raise ValueError(f"{name} must be a finite number")
    return float(value)


def _count(value, name):
    if isinstance(value, bool) or not isinstance(value, Integral) or value < 0:
        raise ValueError(f"{name} must be a non-negative integer")
    return int(value)


def slices_over_budget(old_metrics, new_metrics, budget, slice_counts=None, min_count=1):
    budget = _finite_number(budget, "budget")
    if budget < 0:
        raise ValueError("budget must be non-negative for lower-is-better metrics")
    min_count = _count(min_count, "min_count")
    if min_count == 0:
        raise ValueError("min_count must be positive for a release gate")

    missing_in_new = sorted(set(old_metrics) - set(new_metrics))
    new_only = sorted(set(new_metrics) - set(old_metrics))
    flagged = []
    underpowered = []
    for name in sorted(set(old_metrics) & set(new_metrics)):
        old_value = _finite_number(old_metrics[name], f"{name} old metric")
        new_value = _finite_number(new_metrics[name], f"{name} new metric")
        if slice_counts is not None:
            support = _count(slice_counts.get(name, 0), f"{name} count")
            if support < min_count:
                underpowered.append({"slice": name, "count": support})
                continue
        delta = new_value - old_value
        if delta > budget:
            flagged.append({
                "slice": name,
                "old": old_value,
                "new": new_value,
                "delta": delta,
            })
    return {
        "flagged": flagged,
        "missing_in_new": missing_in_new,
        "new_only": new_only,
        "underpowered": underpowered,
    }