Data Systems
Practice the production skill of improving ASR, TTS, and voice agents from production signals without leaking private audio, overfitting to noisy metrics, or building a labeling process that cannot support release decisions.
Operating Loop
A data flywheel is not "collect more examples." It is a controlled system for turning allowed production signals into candidate examples, labels, eval slices, model changes, release gates, and post-launch monitoring.
Every example used for training or evaluation must have a documented source, consent scope, retention policy, label rubric, split assignment, and model-release lineage. If an example cannot be traced or is not allowed for the intended use, it must not enter the loop.
Active Learning
Active learning should answer a release or research question. For speech systems, useful selection often combines model uncertainty, user impact, audio conditions, language/domain coverage, and known slice gaps.
Low confidence, high partial churn, endpointing retries, rare entities, code-switching, far-field audio, noisy mobile, and correction clusters.
Sampling only the lowest-confidence utterances can overrepresent noise, unsupported languages, or unusable audio. A strong plan stratifies by product slice and adds diversity caps so the label batch improves the eval set instead of becoming a junk drawer.
Pronunciation complaints, long-form instability, number/date errors, emotional tone mismatch, clipping, latency outliers, and speaker consistency.
TTS examples are often text prompts plus generated audio, so the team needs separate labels for text difficulty, pronunciation, prosody, audio artifacts, safety, and latency. Preference alone does not explain what to fix.
ASR entity substitutions, retrieval misses, stale citations, low task completion, repeated clarification, barge-in failures, and unsafe tool attempts.
Select examples with stage tags. Otherwise a failed answer may be mislabeled as an LLM issue when the root cause was ASR, retrieval, stale index metadata, tool policy, or TTS delivery.
Use synthetic utterances, aggregate metrics, redacted text, hashed slice IDs, generated noise conditions, and review-approved fixtures.
Proxies are enough for pipeline logic, CI gates, data contracts, release tooling, and many regression tests. They are not enough to claim final ASR, TTS, or human-experience quality for a real target population.
Labeling Ops
Two annotators disagree about whether "twenty one pilots" should be normalized as a band name or a number phrase. How should the labeling system handle this before the next release?
Add domain-specific normalization rules, capture disagreement as a rubric bug, route ambiguous examples to adjudication, and version the label policy. The release review should report how many labels changed because metric movement may come from policy changes rather than model quality.
Preference scores improved after the annotation vendor changed, but pronunciation complaints increased in production. What happened?
The vendor shift likely changed the measurement. Audit rater demographics, rubric training, gold checks, prompt mix, language coverage, and inter-rater agreement. Add separate pronunciation and named-entity scores so a pleasant voice cannot hide practical failures.
Design labels for a voice assistant that answers policy questions from retrieved documents.
Include ASR entity accuracy, query intent, retrieval recall, source freshness, answer groundedness, citation correctness, refusal correctness, task success, latency, spoken clarity, and safety outcome. Keep stage labels separate from the final end-to-end score.
Release Engineering
A training-data update is a production change. It can improve aggregate metrics while hurting a valuable slice, increasing cost, breaking latency, or changing safety behavior.
Report added, removed, relabeled, and reweighted examples by slice, source, consent scope, and label policy.
Unknown source, missing consent, split leakage, eval-set contamination, unexplained relabeling, or missing slice coverage for the launch population should block until resolved.
Compare aggregate quality, slice quality, calibration, robustness, latency, memory, cost, and safety guardrails.
Data changes can alter output length, beam behavior, retry rate, tool calls, context size, and TTS duration. A model can be more accurate but too expensive or slow for the product budget.
Use shadow traffic, canary cohorts, rollback flags, burn-rate alerts, and post-launch sample audits.
It must include the slices at risk, enough volume for the gate, clear owner response time, predefined rollback triggers, and telemetry that separates model, data, retrieval, and serving failures.
Incident Exercises
Offline WER improved by 3 percent relative, but mobile dictation complaints increased after launch.
Check slice coverage, endpointing and partial churn, UI correction changes, device mix, noisy mobile performance, rare entity recall, canary cohort composition, and label normalization. Roll back or route the affected slice if user harm is clear while preserving aggregate-only incident notes.
The latest training batch came from low-confidence sampling. It improved call-center audio but regressed accented assistant queries.
The sampler likely shifted the training distribution without slice caps or weighting review. Add stratified sampling, per-slice regression budgets, active-learning batch manifests, and an eval gate that protects accented assistant traffic.
A new index refresh improves retrieval recall but increases stale spoken answers.
Add freshness labels, source-effective dates, stale-source eval slices, answer citation checks, and index lineage in the release manifest. The rollback target may be the index, ranker, prompt, or document ingestion job rather than the ASR or LLM model.
Lab
This toy utility scores synthetic aggregate records. It demonstrates the control logic for active-learning queues without storing private audio or transcripts.
Given aggregate candidate metadata, prefer examples with high model uncertainty, high user impact, undercovered slices, and recent regressions. Exclude disallowed consent scopes.
def rank_review_candidates(records, allowed_scopes):
ranked = []
for row in records:
if row["consent_scope"] not in allowed_scopes:
continue
uncertainty = 1.0 - row["confidence"]
impact = min(row["daily_sessions"] / 1000.0, 3.0)
coverage_gap = 1.5 if row["slice_coverage"] < 0.2 else 0.0
regression = 2.0 if row["recent_regression"] else 0.0
privacy_penalty = 1.0 if row.get("sensitive_proxy", False) else 0.0
score = uncertainty + impact + coverage_gap + regression - privacy_penalty
ranked.append((score, row["candidate_id"]))
return [candidate_id for score, candidate_id in sorted(ranked, reverse=True)]Test disallowed consent exclusion, low-coverage boost, regression boost, capped impact, sensitive proxy penalty, tie behavior, empty input, and a case where the highest-uncertainty example is not the best candidate because it has no launch relevance.
Replace hand weights with a review policy owned by data, legal, product, and ML leads. Add quotas by slice, deduplication, human review capacity, lineage IDs, audit logs, retention enforcement, and offline simulation showing that the sampler improves release decisions rather than only increasing label volume.