Module 13
AI engineers need to turn model demos into services with clear latency, memory, throughput, reliability, privacy, and cost behavior. This lesson connects transformer serving to speech workloads.
Mental Model
Queue time, preprocessing, first token or first audio byte, decode time, post-processing, network, and client playback all count.
Interactive speech products are judged by tail latency and jitter. A good mean can hide overloaded queues, cold starts, long-context requests, batch starvation, or a few tenants causing bad p95 and p99 behavior.
Weights, activations, KV cache, audio buffers, tokenizer state, feature tensors, and batching padding all compete for memory.
The cache grows with layers, KV heads, head dimension, sequence length, batch size, and precision. Across many active long conversations, cache memory can rival or exceed the model weights, so schedulers need limits, eviction, paging, truncation, or summarization.
Batching and streaming must be balanced. Larger batches improve accelerator utilization but can delay the first useful response.
Limit batching for conversational ASR, speech-to-speech, and TTS first-audio-byte paths where queue delay is visible. Use larger batches for offline transcription, embeddings, evaluation jobs, and analytics pipelines.
Quantization, pruning, smaller context, decoding shortcuts, and lower vocoder quality can save cost but must be checked by slice.
Track approved aggregate slices such as noisy rooms, far-field microphones, low-volume speech, domain terms, code-switching, long silences, overlapping speech, and rare entity names. Use consented or coarse language/accent and age-band tags only when policy allows them. Global WER can hide severe regressions in these groups.
Capacity Planning
In system design interviews, show that you can estimate before reaching for a benchmark. State assumptions, calculate, then explain which measurements would replace the estimate.
You expect 10,000 concurrent streams at peak. Each stream sends 16 kHz mono audio. The streaming model consumes 0.35 GPU-seconds per audio second after batching, and one GPU should run at no more than 70 percent utilization. Estimate replicas.
Peak demand is 10,000 audio-seconds per wall-clock second. At 0.35 GPU-seconds per audio second, demand is 3,500 GPU-seconds per second. At a 70 percent utilization target, each GPU contributes 0.7 usable GPU-seconds per second. Estimate 3,500 / 0.7 = 5,000 GPUs before optimizations. A strong answer should challenge the assumptions, look for model or batching improvements, regional peak smoothing, smaller models, CPU preprocessing, VAD savings, and offline-vs-real-time separation.
A decoder-only model has 32 layers, 32 KV heads, head dimension 128, fp16 cache, and 8,000 active tokens per request. Estimate KV cache per request before batching overhead.
Cache bytes are layers * tokens * KV tensors * KV heads * head dim * bytes. That is 32 * 8,000 * 2 * 32 * 128 * 2 bytes, about 4.2 GB decimal, or about 3.9 GiB, per request before allocator and batching overhead. The answer should immediately discuss grouped-query attention, shorter context, cache paging, summarization, request limits, and why long conversations can dominate capacity.
Serving Patterns
Use for privacy, offline mode, small models, local wake words, VAD, and personal assistants where device limits are acceptable.
Local serving reduces data exposure and network latency, but makes hardware variability, model updates, telemetry, crash debugging, and battery use harder. Keep a compatibility matrix and a rollback path for each release channel.
Use for large ASR, speech-to-speech, LLM reasoning, neural codecs, and high-quality TTS that exceed device capacity.
Online GPU serving gives centralized control, better utilization, and faster model iteration. It also introduces privacy, network, quota, regional capacity, cold-start, and tail-latency problems.
Use for offline transcription, dataset labeling, eval runs, embedding generation, and backfills where throughput matters most.
Batch jobs should maximize accelerator occupancy, retry safely, checkpoint progress, record model versions, and support idempotent outputs. They should not share fragile capacity with real-time conversational paths unless quotas are strict.
Production Debugging
Model execution is stable, but p99 end-to-end latency doubled. Traces show most of the increase before inference begins.
Check arrival rate, per-tenant spikes, autoscaler lag, batcher thresholds, max queue age, priority classes, stuck workers, retries, and whether shadow or canary traffic is counted twice. Mitigate by reducing batch wait, adding replicas, shedding low-priority traffic, splitting real-time and batch queues, or rolling back the scheduler change.
Short requests pass. Long speech-to-speech sessions start failing with GPU out-of-memory errors after a few turns.
Inspect KV-cache growth, audio-token context, retained hidden states, conversation summarization, max-token limits, request cancellation cleanup, and cache fragmentation. Mitigate with hard context budgets, cache paging, summarization, session eviction, smaller precision, GQA/MQA models, or routing long sessions to a larger pool.
A quantized ASR model cut GPU cost by 30 percent, but customers report more mistakes on names and product SKUs.
Roll back or canary-limit if critical entity error exceeds the launch budget. Compare by approved aggregate slices such as domain terms, SNR, utterance length, and consented coarse language/accent tags when policy allows them. Consider mixed precision for sensitive layers, rescoring, contextual biasing, post-processing, or a cascade where the cheaper model handles easy traffic and uncertain requests go to the baseline.
Lab
Create a one-page report for any ASR, TTS, embedding, or LLM model you can run locally. Do not commit private audio, transcripts, tokens, or credentials.
It connects model quality to operational constraints. A strong report explains what was measured, what was not measured, how the model can fail in production, what blocks launch, and exactly how to roll back or degrade gracefully.