Production Evaluation
Build a fast feedback loop for ASR and TTS products by sampling or flagging live traffic, running audio-aware diagnostic judges, and aggregating likely user pain before a labeled benchmark catches up.
Product Motivation
Offline benchmarks answer whether one system is better than another on curated examples. Production monitoring answers a different question: what felt bad to users today, and what issue should the team fix next?
Use pairwise or rubric evaluations to compare one ASR, TTS, or speech-to-speech system against another on stable slices.
Use it for release decisions, vendor selection, model upgrades, and public claims. Keep prompts versioned, randomize order in pairwise tests, and track judge bias before trusting a leaderboard.
Use flagged recent events, sampled live traffic, corrections, retries, abandons, and explicit negative feedback to find recurring failures.
It does not wait for labels. The judge proposes likely issue classes, severity, and examples. Humans review the top clusters only when an issue becomes important enough to act on.
Event Contract
A production audio judge needs the audio or synthesized output, the relevant text context, versions, and user feedback metadata. It should not require committing private audio, raw user text, or credentials.
{
"id": "event-2026-06-29-0001",
"task": "asr_error",
"source": "consented_production_flagged",
"flag_reason": "user_unsatisfied",
"audio_path": "private/audio/event.wav",
"candidate_text": "redacted recognized transcript or TTS prompt",
"reference_text": "optional redacted user correction or expected text",
"turns": [
{"role": "user", "content": "optional minimized surrounding request"},
{"role": "assistant", "content": "optional minimized system response"}
],
"metadata": {
"app": "LocalWhisperTranscriber",
"app_version": "0.4.2",
"model": "whisper-small",
"latency_ms": 742,
"sample_policy": "user_flagged",
"privacy_tier": "local_only"
}
}Flag the most recent transcription, user correction, command failure, pasted-text undo, repeated dictation, low confidence, or high partial churn.
Flag the most recent spoken output, playback abort, user replay, long first-audio delay, artifact report, text-faithfulness concern, or voice mismatch.
Judge Output
Audio LLM judges are most useful when prompts ask for narrow, schema-validated fields. A vague quality score is hard to debug; issue labels, severity, and short evidence help the team react.
Meaning preservation, entity errors, number/unit mistakes, negation loss, omissions, insertions, speaker-turn confusion, and downstream task impact.
Intelligibility, artifacts, skipped or altered text, unnatural prosody, pacing, voice consistency, clipping, silence, and conversational latency.
Prompt id, prompt version, provider, model, parse status, schema errors, confidence, and whether the case used reference text or audio-direct judging.
Monitoring systems must fail predictably. If a judge omits required diagnostics or invents fields, reports should mark the result as a bounded parse/schema failure instead of silently mixing malformed outputs into top-issue counts.
Aggregation
The goal is not to inspect every transcript. The goal is to cluster likely failures, rank them by severity and frequency, and choose the next model, prompt, decoder, UX, or data fix.
A useful packet says: "Top issue is missing final words in long noisy dictation after release 0.4.2, 37 flagged cases, mostly desktop-class mic, median severity high, examples point to endpointing and final decode overlap. Try longer tail overlap or rollback endpointing change."
Lab
Start from judge results, not raw audio. The lab mirrors a production dashboard where flagged events are already scored and schema-validated.
Given a list of judge result dictionaries, return the top issue groups sorted by weighted impact. Weight each case by severity and user flag strength. Keep only aggregate counts and example IDs.
SEVERITY_WEIGHT = {"low": 1, "medium": 3, "high": 8}
FLAG_WEIGHT = {
"sampled": 1,
"low_confidence": 2,
"user_flagged": 5,
"user_unsatisfied": 8,
}
def top_audio_issues(results, limit=5):
if limit < 1:
return []
groups = {}
seen_event_ids = set()
for row in results:
if not isinstance(row, dict):
continue
diagnostics = row.get("diagnostics", {})
metadata = row.get("metadata", {})
if not isinstance(diagnostics, dict) or not isinstance(metadata, dict):
continue
event_id = row.get("id")
if event_id in seen_event_ids:
continue
issue = diagnostics.get("primary_issue")
severity = diagnostics.get("severity", "medium")
flag = metadata.get("flag_reason", "sampled")
if (
not isinstance(event_id, str)
or not isinstance(issue, str)
or severity not in SEVERITY_WEIGHT
or flag not in FLAG_WEIGHT
):
continue
seen_event_ids.add(event_id)
weight = SEVERITY_WEIGHT[severity] * FLAG_WEIGHT[flag]
bucket = groups.setdefault(issue, {
"issue": issue,
"count": 0,
"impact": 0,
"examples": [],
"models": {},
"versions": {},
})
bucket["count"] += 1
bucket["impact"] += weight
bucket["examples"] = (bucket["examples"] + [event_id])[:3]
model = metadata.get("model", "unknown")
if not isinstance(model, str):
model = "unknown"
bucket["models"][model] = bucket["models"].get(model, 0) + 1
version = metadata.get("app_version", "unknown")
if not isinstance(version, str):
version = "unknown"
bucket["versions"][version] = bucket["versions"].get(version, 0) + 1
return sorted(groups.values(), key=lambda item: (-item["impact"], -item["count"]))[:limit]Test that user-unsatisfied high-severity cases outrank many sampled low-severity cases, malformed judge outputs are excluded before aggregation, duplicate event IDs do not inflate impact, example IDs are capped, and model/version breakdowns stay aggregate-only.
Reading Links