Data Systems
Practice the production skill of improving ASR, TTS, and voice agents from production signals without leaking private audio, overfitting to noisy metrics, or building a labeling process that cannot support release decisions.
Operating Loop
A data flywheel is not "collect more examples." It is a controlled system for turning allowed production signals into candidate examples, labels, eval slices, model changes, release gates, and post-launch monitoring.
Every example used for training or evaluation must have a documented source, consent scope, retention policy, label rubric, split assignment, and model-release lineage. If an example cannot be traced or is not allowed for the intended use, it must not enter the loop. A newly labeled active-learning batch should not silently update both the training set and the eval set, because that turns release metrics into a memory test instead of an independent quality check.
Active Learning
Active learning should answer a release or research question. For speech systems, useful selection often combines model uncertainty, user impact, audio conditions, language/domain coverage, and known slice gaps.
Low confidence, high partial churn, endpointing retries, rare entities, code-switching, far-field audio, noisy mobile, and correction clusters.
Sampling only the lowest-confidence utterances can overrepresent noise, unsupported languages, or unusable audio. A strong plan stratifies by product slice and adds diversity caps so the label batch improves the eval set instead of becoming a junk drawer.
Pronunciation complaints, long-form instability, number/date errors, emotional tone mismatch, clipping, latency outliers, and speaker consistency.
TTS examples are often text prompts plus generated audio, so the team needs separate labels for text difficulty, pronunciation, prosody, audio artifacts, safety, and latency. Preference alone does not explain what to fix.
ASR entity substitutions, retrieval misses, stale citations, low task completion, repeated clarification, barge-in failures, and unsafe tool attempts.
Select examples with stage tags. Otherwise a failed answer may be mislabeled as an LLM issue when the root cause was ASR, retrieval, stale index metadata, tool policy, or TTS delivery.
Use synthetic utterances, aggregate metrics, redacted text, approved coarse slice buckets, generated noise conditions, and review-approved fixtures.
Proxies are enough for pipeline logic, CI gates, data contracts, release tooling, and many regression tests. They are not enough to claim final ASR, TTS, or human-experience quality for a real target population.
Labeling Ops
Two annotators disagree about whether "twenty one pilots" should be normalized as a band name or a number phrase. How should the labeling system handle this before the next release?
Add domain-specific normalization rules, capture disagreement as a rubric bug, route ambiguous examples to adjudication, and version the label policy. The release review should report how many labels changed because metric movement may come from policy changes rather than model quality.
Preference scores improved after the annotation vendor changed, but pronunciation complaints increased in production. What happened?
The vendor shift likely changed the measurement. Audit rater coverage metadata approved for evaluation, rubric training, gold checks, prompt mix, language coverage, and inter-rater agreement. Add separate pronunciation and named-entity scores so a pleasant voice cannot hide practical failures.
Design labels for a voice assistant that answers policy questions from retrieved documents.
Include ASR entity accuracy, query intent, retrieval recall, source freshness, answer groundedness, citation correctness, refusal correctness, task success, latency, spoken clarity, and safety outcome. Keep stage labels separate from the final end-to-end score.
Release Engineering
A training-data update is a production change. It can improve aggregate metrics while hurting a valuable slice, increasing cost, breaking latency, or changing safety behavior.
Report added, removed, relabeled, and reweighted examples by slice, source, consent scope, and label policy; bucket sparse or sensitive slices before sharing metrics.
Unknown source, missing consent, split leakage, eval-set contamination, unexplained relabeling, or missing slice coverage for the launch population should block until resolved.
Compare aggregate quality, slice quality, calibration, robustness, latency, memory, cost, and safety guardrails.
Data changes can alter output length, beam behavior, retry rate, tool calls, context size, and TTS duration. A model can be more accurate but too expensive or slow for the product budget.
Use shadow traffic, canary cohorts, rollback flags, burn-rate alerts, and post-launch sample audits.
It must include the slices at risk, enough volume for the gate, clear owner response time, predefined rollback triggers, and telemetry that separates model, data, retrieval, and serving failures.
Incident Exercises
Offline WER improved by 3 percent relative, but mobile dictation complaints increased after launch.
Check slice coverage, endpointing and partial churn, UI correction changes, device mix, noisy mobile performance, rare entity recall, canary cohort composition, and label normalization. Roll back or route the affected slice if user harm is clear while preserving aggregate-only incident notes.
The latest training batch came from low-confidence sampling. It improved call-center audio but regressed assistant queries with pronunciation variation.
The sampler likely shifted the training distribution without slice caps or weighting review. Add stratified sampling, approved aggregate or consented pronunciation-variation slices, per-slice regression budgets, active-learning batch manifests, and an eval gate that protects launch-critical assistant traffic.
A new index refresh improves retrieval recall but increases stale spoken answers.
Add freshness labels, source-effective dates, stale-source eval slices, answer citation checks, and index lineage in the release manifest. The rollback target may be the index, ranker, prompt, or document ingestion job rather than the ASR or LLM model.
Lab
This toy utility scores synthetic aggregate records. It demonstrates the control logic for active-learning queues without storing private audio or transcripts.
Given aggregate candidate metadata, prefer examples with high model uncertainty, high user impact, undercovered slices, and recent regressions. Exclude disallowed consent scopes and cap each slice so one noisy cohort cannot consume the whole review batch.
import math
def finite_number(value, name):
if isinstance(value, bool) or not isinstance(value, (int, float)):
raise ValueError(f"{name} must be numeric")
if not math.isfinite(value):
raise ValueError(f"{name} must be finite")
return float(value)
def rank_review_candidates(records, allowed_scopes, max_per_slice=2, min_slice_count=20):
if not isinstance(max_per_slice, int) or max_per_slice <= 0:
raise ValueError("max_per_slice must be a positive integer")
if not isinstance(min_slice_count, int) or min_slice_count <= 0:
raise ValueError("min_slice_count must be a positive integer")
ranked = []
for row in records:
if row["consent_scope"] not in allowed_scopes:
continue
slice_id = row["slice_id"]
if not isinstance(slice_id, str) or not slice_id:
raise ValueError("slice_id must be a non-empty string")
confidence = finite_number(row["confidence"], "confidence")
daily_sessions = finite_number(row["daily_sessions"], "daily_sessions")
slice_coverage = finite_number(row["slice_coverage"], "slice_coverage")
if not 0.0 <= confidence <= 1.0:
raise ValueError("confidence must be between 0 and 1")
if daily_sessions < 0:
raise ValueError("daily_sessions must be non-negative")
if not 0.0 <= slice_coverage <= 1.0:
raise ValueError("slice_coverage must be between 0 and 1")
slice_count = finite_number(row.get("slice_count", min_slice_count), "slice_count")
if slice_count < 0:
raise ValueError("slice_count must be non-negative")
if not isinstance(row["recent_regression"], bool):
raise ValueError("recent_regression must be boolean")
if not isinstance(row.get("sensitive_proxy", False), bool):
raise ValueError("sensitive_proxy must be boolean")
if row.get("sensitive_proxy", False) and slice_count < min_slice_count:
slice_id = "bucketed_sensitive_proxy"
uncertainty = 1.0 - confidence
impact = min(daily_sessions / 1000.0, 3.0)
coverage_gap = 1.5 if slice_coverage < 0.2 else 0.0
regression = 2.0 if row["recent_regression"] else 0.0
privacy_penalty = 1.0 if row.get("sensitive_proxy", False) else 0.0
score = uncertainty + impact + coverage_gap + regression - privacy_penalty
ranked.append((score, row["candidate_id"], slice_id))
ranked.sort(key=lambda item: (-item[0], item[1]))
selected = []
per_slice = {}
for score, candidate_id, slice_id in ranked:
if per_slice.get(slice_id, 0) >= max_per_slice:
continue
selected.append(candidate_id)
per_slice[slice_id] = per_slice.get(slice_id, 0) + 1
return selectedTest disallowed consent exclusion, low-coverage boost, regression boost, capped impact, sensitive proxy penalty, tie behavior, empty input, invalid confidence or coverage ranges, negative traffic, non-finite numeric inputs, missing slice IDs, sparse sensitive slice bucketing, and per-slice caps. Include a case where the highest-uncertainty example is not the best candidate because it has no launch relevance.
Replace hand weights with a review policy owned by data, legal, product, and ML leads. Add quotas by slice, deduplication, human review capacity, lineage IDs, audit logs, retention enforcement, and offline simulation showing that the sampler improves release decisions rather than only increasing label volume.