Module 26
ML engineers are judged by how safely they turn model improvements into product changes. This lesson covers release pipelines, model registry contracts, CI gates, canaries, rollback, and production debugging for audio-text systems.
Release Pipeline
A strong release pipeline starts before deployment. It preserves the training data manifest, model artifact, feature code, tokenizer or codec version, decoding configuration, evaluation pack, and serving image. Without those links, rollback and postmortems become guesswork.
The artifact depends on feature extraction, tokenizer or codec version, decoding parameters, runtime libraries, safety filters, and routing rules. Rolling back only weights can leave the system in a mixed state where outputs are not reproducible.
Contracts
Registry entries should say what the artifact is allowed to do and how it must be evaluated before serving.
Include model ID, version, owner, intended task, base model, data manifest, training code SHA, feature contract, input schema, output schema, eval report, safety report, serving image, resource envelope, rollback target, and expiration or review date.
Serving contracts make client behavior stable even when the model changes behind the endpoint.
Specify audio format, sample rate, chunk size, timestamps, partial versus final transcript semantics, cancellation, barge-in, language hints, speaker metadata policy, redaction behavior, error codes, latency SLOs, and compatibility with previous client versions.
Research trained an ASR model with 16 kHz mono log-mel features. A production optimization silently resamples some mobile traffic to 12 kHz before feature extraction. What can break?
The model may see shifted frequency content and degraded phoneme evidence, especially for sibilants, noise, and accents. The release should fail a contract test before canary. A strong answer asks for sample-rate metadata, feature checksums, slice WER, client version splits, and replay fixtures that verify the exact production feature path.
Rollback
Rollback must be fast, rehearsed, and observable. It should restore user-visible behavior, not just flip a deployment back to an old image.
A new streaming decoder improves final WER but raises partial transcript churn during noisy calls.
Route affected noisy-call slices back to the previous decoder, keep global canary if unaffected slices are healthy, verify client committed-prefix behavior, and watch first partial latency, final WER, partial churn, and command success. If routing is not trusted, roll back the whole decoder release.
A new voice is more natural, but p99 first-audio-byte breaches the spoken-turn SLO during peak traffic.
Restore the old voice or reduce traffic to the new voice, pre-warm only if capacity math supports it, and confirm user-visible turn latency recovers. Track p50, p95, p99, queue age, GPU memory, retry rate, and cost per successful turn.
Coding Lab
Use synthetic metadata and aggregate metrics only. These functions mirror the checks an experienced engineer should expect in CI or deployment automation.
Given a registry entry, return missing fields and block release when critical metadata is absent.
REQUIRED_FIELDS = {
"model_id",
"version",
"owner",
"task",
"data_manifest",
"training_code_sha",
"feature_contract",
"eval_report",
"serving_image",
"rollback_target",
}
def validate_registry_entry(entry):
missing = sorted(field for field in REQUIRED_FIELDS if not entry.get(field))
return {
"release_blocked": bool(missing),
"missing_fields": missing,
}The invariant is that every promoted model must be traceable and reversible. The common mistake is validating only model weights and ignoring feature, eval, and rollback metadata.
Decide whether a canary can promote based on slice metrics. Lower WER, lower latency, and lower cost are better.
def canary_promotion(metrics):
blockers = []
for name, values in metrics["slices"].items():
if values["wer_relative_delta"] > 0.02:
blockers.append(f"{name}:wer")
if values["p95_latency_ms_delta"] > 75:
blockers.append(f"{name}:latency")
if metrics["privacy_errors"] > 0:
blockers.append("privacy")
if metrics["unit_cost_relative_delta"] > 0.15:
blockers.append("unit_cost")
if blockers:
return {"decision": "hold", "blockers": blockers}
return {"decision": "promote", "blockers": []}Average quality improvement is not enough. A canary should protect critical slices, privacy, latency, and unit economics before promotion.
Given active releases and health state, choose the rollback target for a model family.
def rollback_target(releases, family):
candidates = [
release for release in releases
if release["family"] == family and release["state"] == "stable"
]
if not candidates:
raise ValueError(f"no stable rollback target for {family}")
candidates.sort(key=lambda release: release["promoted_at"], reverse=True)
return {
"model_id": candidates[0]["model_id"],
"version": candidates[0]["version"],
"serving_image": candidates[0]["serving_image"],
}The target must be a known stable serving bundle, not merely the previous commit. A release system records enough metadata to restore behavior quickly.
Advanced Prompts
Design CI gates for a streaming ASR model used in dictation and voice commands. Include tests, artifacts, release gates, and rollback.
Cover unit tests for feature extraction, schema compatibility, sanitized replay fixtures, WER/CER slices, command intent recall, partial churn, endpointing, latency, memory, privacy logging, canary thresholds, model registry metadata, and a tested rollback target. Dictation and commands need separate quality gates.
A TTS release passed offline evaluation but caused a production outage because GPU memory rose under concurrent traffic. What should change?
Add load tests with realistic concurrency, queue age, warmup, voice mix, long utterances, memory fragmentation, retry behavior, and autoscaling. Require serving-image benchmarks in CI, capacity signoff, narrow canaries, and rollback verification before the next release.
Researchers want weekly model releases, but production teams worry about regressions. How do you speed up safely?
Build a paved release path with registry templates, automated eval packs, synthetic replay, shadow traffic, slice dashboards, progressive canaries, fast rollback, and clear ownership. Speed comes from removing manual ambiguity, not skipping gates.