Production Serving

Prefill work is parallel over prompt tokens and can dominate time for long contexts. Decode work adds one token per active sequence and often becomes memory-bandwidth bound because every step reads cached keys and values. Continuous batching keeps the device busy by adding and removing requests at step boundaries instead of waiting for a whole static batch to finish.

serving_budget = {
    "prefill_tokens_per_second": measure_prompt_throughput(),
    "decode_tokens_per_second": measure_generation_throughput(),
    "max_active_tokens": floor(kv_memory_budget / kv_bytes_per_token),
    "tail_latency_guard": p95_ms <= target_ms,
}

Latency Drill: Tune TTFT And ITL Separately

Do not collapse serving latency into one average. Time-to-first-token mostly exposes queueing and prefill pressure, while inter-token latency exposes decode-step contention. vLLM's chunked prefill guidance frames max_num_batched_tokens as a throughput and latency knob: smaller token budgets can protect decode latency, while larger budgets admit more prefill work per iteration. TensorRT-LLM's in-flight batching uses the same core idea of changing the active request set during the generation loop instead of waiting for a static batch to drain.

1. Measure separately: record queue time, prefill time, first-token time, and per-token decode time.
2. Stress long prompts: mix short chat requests with long-context requests and watch whether decode tokens stall behind prefill chunks.
3. Change one knob: sweep max batched tokens or partial-prefill limits before changing model, quantization, or hardware.

Scheduler Drill: Decode-First Continuous Batching

A serving scheduler is choosing which token work enters each forward pass. PagedAttention makes variable-length requests cheaper to pack by storing KV blocks non-contiguously, while continuous batching lets completed streams leave and new streams enter between decode steps. vLLM's optimization guide adds chunked prefill so long prompts can be split into smaller token chunks and batched with decode work; Hugging Face TGI exposes the same production goal by combining continuous batching, streaming, Flash Attention, Paged Attention, and KV caching. A practical policy is to protect streaming decode first, then spend remaining token budget on prefill chunks.

def schedule_step(waiting_prefills, running_decodes, token_budget):
    batch = []
    for request in running_decodes:
        if token_budget == 0:
            break
        batch.append(("decode_one_token", request))
        token_budget -= 1

    for request in waiting_prefills:
        if token_budget == 0:
            break
        chunk = min(request.remaining_prompt_tokens, token_budget)
        batch.append(("prefill_chunk", request, chunk))
        token_budget -= chunk

    return batch

1. Preserve streams: reserve decode slots for already-streaming requests before admitting large prompt chunks.
2. Bound prompt chunks: tune the batched-token budget so one long context improves TTFT without starving ITL for active users.
3. Pair with paged cache: admission still needs enough free KV blocks for each promised prompt and output token after block rounding.
4. Watch both counters: a scheduler win should show prompt-token throughput without a p95 ITL regression for running streams.

Batching Drill: Keep The Active Batch Shape Honest

ORCA introduced iteration-level scheduling and selective batching to avoid treating a whole generation request as one indivisible batch item. Current serving engines expose the same operational lesson: TensorRT-LLM describes in-flight batching as adding newly arrived requests and returning completed ones at each generation-loop iteration, while Hugging Face TGI advertises continuous batching with streaming. A practical scheduler should therefore account for each request phase, current cache length, promised output budget, and adapter or prefix-cache constraints before it packs the next step.

def build_iteration_batch(waiting, running, limits):
    candidates = [r for r in running if r.needs_next_token()]
    candidates += pick_prefill_chunks(waiting, limits.prefill_token_budget)

    batch = []
    for request in sorted(candidates, key=batching_priority):
        if not fits_kv_blocks(request, batch, limits.free_kv_blocks):
            continue
        if not compatible_adapter_set(request, batch, limits.max_loras):
            continue
        if not fits_token_budget(request, batch, limits.max_batched_tokens):
            continue
        batch.append(request)
    return batch

1. Schedule iterations: let completed streams leave and late arrivals enter between token steps instead of pinning one static batch until every request finishes.
2. Batch by compatible work: group requests that can share model weights, adapter limits, cache policy, and kernel assumptions before chasing a larger batch size.
3. Protect short streams: track whether long prompts, high max-output promises, or cache-heavy adapters are crowding out low-latency decode work.
4. Report the shape: log active sequences, prefill tokens, decode tokens, KV blocks, adapter ids, and dropped candidates for each scheduler iteration.

Prefill/decode disaggregation runs prompt processing and token generation on separate workers, then transfers the produced KV cache to the decode side. The idea is attractive when long prompts and streaming decode interfere with each other, or when the two phases should scale on different hardware. It is not a free optimization: the service now owns KV transfer latency, connector reliability, routing, autoscaling for two pools, and harder failure recovery.

use_disaggregation = (
    p95_ttft_is_prefill_bound
    and decode_itl_regresses_during_prompt_spikes
    and kv_transfer_ms < saved_queueing_ms
    and ops_can_run_two_worker_pools
)

1. Prove interference first: compare TTFT and inter-token latency under mixed short-chat and long-prompt load before changing topology.
2. Account for KV movement: transferred cache blocks must arrive before decode can continue, so network and connector overhead belong in the latency budget.
3. Scale asymmetrically: size prefill workers from prompt-token demand and decode workers from active-stream and output-token demand.
4. Keep a fallback path: retain an aggregated serving profile for incidents, small deployments, or traffic shapes where transfer overhead dominates.

Readiness Drill: Prove KV Transfer Is Faster Than Waiting

Disaggregated serving should be justified by phase-specific evidence, not by topology fashion. vLLM's disaggregated prefill docs describe separate prefill and decode instances with KV-cache transfer between them; Ray Serve frames the same split as a way to scale prefill-heavy and decode-heavy workers independently; TensorRT-LLM's writeup calls out direct KV-cache movement and layout mapping when the two phases use different parallel strategies. Before launch, benchmark the transfer path as a first-class dependency.

disagg_readiness = {
    "baseline": "single_pool_continuous_batching",
    "candidate": "prefill_pool + decode_pool + kv_transfer",
    "required_win": p95_ttft_saved_ms > p95_kv_transfer_ms + routing_ms,
    "guardrail": p95_itl_candidate <= p95_itl_baseline,
    "fallback": "route all traffic back to aggregated workers",
}

1. Measure the link: log KV-transfer bytes, transfer latency, connector errors, and decode wait time after prefill completes.
2. Test mixed load: replay short chat, long RAG, and streaming-heavy traffic together so the candidate proves it reduces phase interference.
3. Match parallel plans: record tensor and pipeline parallelism for each pool because cache layout conversion can become part of the critical path.
4. Keep routing reversible: a failed connector, cold prefill pool, or bad cache hit pattern should have a tested aggregated fallback profile.

Sources: vLLM disaggregated prefilling, Ray Serve prefill/decode disaggregation, TensorRT-LLM disaggregated serving.

Mixture-of-experts models activate only a subset of expert weights per token, but serving them is not just dense-model serving with fewer FLOPs. The router creates per-token expert traffic, expert parallelism spreads expert weights across devices, and all-to-all communication can become the new bottleneck when routing is skewed. DeepSeek-V3 reports a large MoE model with far fewer activated parameters per token than total parameters, while vLLM and TensorRT-LLM both treat expert parallelism as a first-class deployment strategy for MoE inference.

moe_serving_gate = {
    "model": "moe_checkpoint@revision",
    "parallelism": ["data_parallel_attention", "expert_parallel_mlp"],
    "routing_metrics": ["tokens_per_expert", "dropped_tokens", "expert_queue_ms"],
    "communication_metrics": ["all_to_all_ms", "bytes_per_step"],
    "rollback": "dense_or_smaller_moe_profile",
}

1. Measure expert skew: log routed tokens per expert and per layer; average activation count can hide one hot expert stalling a step.
2. Account for communication: expert parallelism saves local weight bandwidth only if all-to-all transfer and synchronization stay inside the latency budget.
3. Batch by compatibility: model revision, adapter state, context length, and expert-parallel group all affect whether requests can share an efficient iteration.
4. Keep load balance visible: compare throughput, p95 TTFT, p95 ITL, and expert utilization before changing router settings or parallel topology.

Sources: DeepSeek-V3 technical report, vLLM expert parallel deployment, TensorRT-LLM expert parallelism.

A production scheduler should reject, queue, or shorten work before the KV cache is exhausted. Paged cache systems expose capacity in blocks, not just bytes, so a practical admission check estimates the prompt tokens already admitted, the maximum new tokens promised to each active request, and the block slack left after rounding each sequence up to the cache block size.

def can_admit(request, active, free_kv_tokens, block_size):
    promised = request.prompt_tokens + request.max_new_tokens
    active_promised = sum(r.cache_len + r.remaining_budget for r in active)
    rounded = ceil_to_block(promised + active_promised, block_size)
    return rounded <= free_kv_tokens

1. Use promises, not averages: reserve against each request's max generation budget, then release unused cache as requests finish.
2. Track block waste: many short requests can waste a meaningful number of tail slots when each sequence rounds to a page or block boundary.
3. Fail predictably: when capacity is tight, prefer a bounded queue, a shorter max-output offer, or a controlled 429-style error over worker OOM.

Admission Drill: Round Each Sequence Before Summing

Paged KV caches allocate fixed-size blocks to each live sequence, so tail slack is paid per request. The vLLM PagedAttention design describes KV cache as blocks with a fixed number of tokens, and TensorRT-LLM exposes paged-cache settings and token-block reuse. A scheduler that sums raw promised tokens first and rounds once can undercount many short requests; round every request's current and promised length to blocks before comparing against free capacity.

def blocks_needed(tokens, block_size):
    return (tokens + block_size - 1) // block_size

def promised_blocks(request, block_size):
    promised_tokens = request.cache_len + request.remaining_prompt + request.max_new_tokens
    return blocks_needed(promised_tokens, block_size)

def can_admit_by_blocks(candidate, active, free_blocks, block_size):
    needed = promised_blocks(candidate, block_size)
    needed += sum(promised_blocks(r, block_size) for r in active)
    return needed <= free_blocks

1. Round per sequence: ten requests with one promised token can consume ten final blocks, not one shared block.
2. Reserve future output: count admitted maximum-new-token promises so a stream does not OOM after users are already receiving tokens.
3. Release on finish: return unused promised blocks when a request stops early, then let the scheduler admit queued work.
4. Report slack: log raw tokens, allocated blocks, and unused tail slots so short-request traffic does not hide wasted capacity.

Implementation Drill: Account For Memory Before Load Testing

A serving profile should reserve GPU memory in buckets before the first traffic run: model weights, KV cache, temporary activations, framework overhead, and fragmentation headroom. Hugging Face documents different cache classes with different memory behavior, while TensorRT-LLM exposes KV-cache sizing and offload controls such as GPU memory fraction and host cache size. Treat those knobs as admission-control inputs, not just runtime defaults.

def estimate_decode_capacity(gpu_bytes, weights_bytes, runtime_overhead,
                             kv_bytes_per_token, safety=0.15):
    usable = gpu_bytes * (1 - safety) - weights_bytes - runtime_overhead
    if usable <= 0:
        return 0
    return floor(usable / kv_bytes_per_token)

active_token_budget = estimate_decode_capacity(
    gpu_bytes=80 * 2**30,
    weights_bytes=model_resident_bytes,
    runtime_overhead=activation_and_allocator_bytes,
    kv_bytes_per_token=2 * layers * kv_heads * head_dim * cache_dtype_bytes,
)

1. Keep buckets separate: shrinking weights with int4 or FP8 does not automatically shrink the KV cache or temporary activation peak.
2. Budget the promise: reserve for admitted prompt tokens plus maximum generated tokens, then release unused cache as streams finish.
3. Leave headroom: allocator fragmentation, CUDA graphs, speculative decoding buffers, and framework workspaces can turn a theoretical fit into an OOM.
4. Recompute after policy changes: static cache, dynamic cache, quantized cache, offloaded cache, tensor parallelism, and LoRA adapters all change the memory ledger.

Admission Drill: Budget Multi-LoRA Traffic Explicitly

Multi-adapter serving lets one base model handle many fine-tuned tasks, but it is not free capacity. vLLM's LoRA docs expose limits such as max_loras for the number of adapters in one batch and max_lora_rank for the largest supported rank; Ray Serve's multi-LoRA guide forwards the same knobs to the vLLM engine; Hugging Face TGI describes loading multiple PEFT-compatible LoRA models at startup. Treat adapter id and rank as scheduler dimensions alongside prompt tokens, because mixing too many adapters can reduce batching efficiency and reserve memory for ranks the traffic rarely uses.

def can_batch_with_loras(candidate, batch, max_loras, max_lora_rank):
    adapter_ids = {r.adapter_id for r in batch if r.adapter_id is not None}
    if candidate.adapter_id is not None:
        adapter_ids.add(candidate.adapter_id)
    if len(adapter_ids) > max_loras:
        return False
    if candidate.lora_rank > max_lora_rank:
        return False
    return same_base_model(candidate, batch) and fits_kv_budget(candidate, batch)

1. Group by base model first: LoRA adapters share a resident base model, so requests for different base checkpoints belong in separate capacity pools.
2. Keep rank tight: set the maximum served rank to the real adapter set instead of a generous placeholder that wastes memory.
3. Limit active adapters: cap the number of distinct adapters in a batch or replica, then route hot adapters to warm workers before falling back to load-on-demand.
4. Measure slices: report TTFT, ITL, cache pressure, adapter load time, and batch size by adapter id so one busy tenant does not hide global regressions.

KV cache is usually the first hard memory limit for long-context serving. Paged cache managers reduce waste from variable-length requests, prefix caching reuses prompt blocks when many requests share the same prefix, and cache quantization can lower memory use at the cost of extra implementation and quality checks.

1. Estimate: compute KV bytes per token for the target model, dtype, layers, KV heads, and batch.
2. Measure: benchmark prefill and decode separately under realistic prompt and output lengths.
3. Constrain: set max model length, max batched tokens, and queue limits from memory and p95 latency data.
4. Optimize: test paged cache, prefix caching, chunked prefill, quantization, and tensor parallelism one at a time.

Implementation Drill: Treat Paged KV Cache Like A Block Table

The PagedAttention paper and vLLM design docs frame KV cache as fixed-size blocks that may live in non-contiguous memory. The scheduler should reason about a logical sequence of cache blocks for each request, while the runtime maps those logical blocks to physical cache blocks. That separation is what lets variable-length requests share a GPU without reserving one giant contiguous allocation per stream.

def append_token(block_table, free_blocks, block_size, token_index):
    logical_block = token_index // block_size
    offset = token_index % block_size
    if logical_block == len(block_table):
        block_table.append(free_blocks.pop())
    physical_block = block_table[logical_block]
    return physical_block, offset

1. Track logical length: the request owns a token count and a list of logical cache blocks, not one contiguous tensor sized for max context.
2. Allocate on demand: add a physical block only when generation crosses a block boundary, then return blocks as requests finish.
3. Budget tail waste: the final block of each live sequence may be partially empty, so short streams can still consume more blocks than raw token count suggests.
4. Keep kernels aware: attention kernels must follow the block table when reading old keys and values, because adjacent logical tokens may not be adjacent in physical memory.

Implementation Drill: Prefix Cache Keys Need A Trust Boundary

Prefix caching is a serving optimization, but the cache key is a product decision. vLLM's design maps processed KV blocks through hashes derived from the prefix history, current block tokens, and optional extra fields such as LoRA identifiers or multimodal hashes. That makes repeated system prompts cheap, but it also means multi-tenant services should decide whether tenants, products, or security domains share a cache namespace.

prefix_cache_key = hash((
    tenant_or_policy_salt,
    parent_block_hash,
    tuple(block_token_ids),
    model_revision,
    adapter_id,
    decoding_contract_version,
))

1. Salt deliberately: add a tenant or policy salt when cache reuse across customers would be surprising, even if the raw token prefix matches.
2. Hash strongly when needed: use a collision-resistant hash mode for untrusted or cross-tenant traffic rather than assuming the fastest hash is good enough.
3. Invalidate by contract: include model revision, adapter id, tokenizer version, and schema/tool contract changes in the cache namespace.
4. Measure the tradeoff: isolation can reduce hit rate, so compare TTFT savings, collision risk, and memory pressure under realistic prompt templates.

Security Drill: Pick The Prefix-Cache Hash Mode Deliberately

Prefix-cache keys are not just performance metadata. vLLM's current design uses block hashes that include parent hash, block token ids, and extra fields such as LoRA ids, multimodal hashes, and cache salts; its docs also call out sha256 as the default hash algorithm in v0.11, with sha256_cbor available when deterministic cross-environment serialization matters. The earlier vLLM advisory on predictable Python hash collisions is a good interview reminder: untrusted multi-tenant cache reuse needs an explicit collision and namespace policy, not just a fast hash table.

prefix_hash_policy = {
    "single_tenant_lab": "fast_hash_ok_with_collision_tests",
    "shared_service": "sha256_plus_cache_salt",
    "multi_runtime_fleet": "sha256_cbor_for_reproducible_keys",
    "log": ["hash_algo", "cache_salt_present", "model_revision", "adapter_id"],
}

1. Choose by trust boundary: use collision-resistant hashing and cache salts when one tenant could influence another tenant's reusable prefix space.
2. Choose by reproducibility: prefer canonical serialization when cache keys must be compared across processes, languages, or service versions.
3. Measure the cost: benchmark hash time against saved prefill time; a small hashing overhead can be worth it when it prevents unsafe reuse.
4. Audit the key fields: every field that changes hidden model state, including multimodal inputs and adapters, belongs in the cache key or namespace.

Implementation Drill: Put Stable Prompt Parts First

Prefix reuse only helps when requests share the same initial token sequence. OpenAI's prompt-caching guide recommends placing static instructions, examples, images, and tools before user-specific content, and reports cache-hit tokens through cached_tokens. vLLM's automatic prefix caching works at the KV-cache block level, so partial matches are rounded by block boundaries and by the server's cache namespace. In both cases, prompt layout is a serving concern as much as a prompting concern.

prompt = [
    stable_system_instructions,
    stable_tool_schemas,
    stable_few_shot_examples,
    tenant_or_task_context,
    latest_user_request,
]

1. Stabilize the prefix: keep shared instructions and schemas byte-for-byte stable across requests before appending volatile user data.
2. Track hits: log provider cache fields such as cached_tokens or engine metrics such as prefix-cache hit rate alongside TTFT.
3. Do not overpromise: caching can reduce prefill cost and latency, but it does not reduce output-token work or fix slow decode.
4. Guard privacy: combine prompt layout with cache namespace or salt policy when identical prefixes should not be shared across tenants.

Implementation Drill: Route Repeated Prefixes To The Cache That Knows Them

Prefix reuse is not only a local memory trick. SGLang describes its runtime as using RadixAttention for prefix caching, and its tuning docs expose a longest-prefix-match scheduling policy for workloads with many shared prefixes. The RadixAttention paper generalizes the idea by retaining KV cache in a radix tree so multi-call programs, self-consistency branches, and agent workflows can reuse common prefixes across generation calls. A production router should therefore preserve prompt-template stability and, when the service is sharded, prefer workers that already hold the longest useful prefix.

def choose_worker(request, workers):
    token_ids = tokenize_with_pinned_template(request)
    candidates = []
    for worker in workers:
        match = worker.prefix_index.longest_prefix_match(
            token_ids,
            namespace=request.cache_namespace,
        )
        candidates.append((match.cached_tokens, -worker.queue_depth, worker))
    return max(candidates)[2]

1. Pin the template: chat template, tool-schema ordering, tokenizer version, and model revision must be stable before cache hits are meaningful.
2. Route by reusable tokens: prefer a warm worker only when the saved prefill tokens outweigh extra queueing or cross-region latency.
3. Evict deliberately: track protected active cache separately from evictable completed-prefix cache, then compare LRU-style policy with workload-specific priority rules.
4. Validate exactness: prefix reuse should skip redundant prefill compute without changing logits, so cache keys must include every factor that changes model states.

Lab Prompt: Estimate Prefix-Cache Reuse From Block Hashes

Before enabling a fleet-wide prefix-cache policy, simulate the hit shape from real prompt templates. vLLM's prefix-caching design maps full KV blocks through a hash of the parent block, current block tokens, and extra cache-key fields; the PagedAttention paper explains why this block-table indirection lets requests share cache without requiring contiguous memory. A useful dry run is to tokenize production-like prompts, split them into cache blocks, and count how many complete prefix blocks would be reused under each namespace policy.

def prefix_block_keys(token_ids, block_size, namespace):
    keys = []
    parent = namespace
    for start in range(0, len(token_ids), block_size):
        block = tuple(token_ids[start:start + block_size])
        if len(block) < block_size:
            break
        parent = hash((parent, block))
        keys.append(parent)
    return keys

hit_rate = reused_complete_blocks / requested_complete_blocks

1. Use token ids: byte-identical text is not the real boundary; tokenizer version and chat template decide the actual prefix blocks.
2. Ignore partial tails: incomplete final blocks may not be reusable in the same way as full KV blocks, so count them separately.
3. Compare namespaces: run the same trace with global, tenant, adapter, and model-revision salts to see the privacy and hit-rate tradeoff.
4. Report saved prefill: translate reused blocks into avoided prompt tokens, then compare that estimate with TTFT and prefix-cache-hit metrics after rollout.

Readiness Drill: Prove Prefix Reuse Is Reachable

Enabling prefix caching is not the same as getting cache hits. vLLM's automatic prefix caching speeds requests that share an existing prefix, but its documented limit is that it only saves prefill work and does not help decode-heavy requests. TensorRT-LLM's KV cache reuse docs add practical reachability constraints: reusable state may not be available until the earlier request terminates, LRU eviction can remove blocks under memory pressure, and only full cache blocks are shared. A launch test should therefore prove that repeated prompts arrive after reusable blocks exist and before those blocks are evicted.

prefix_reuse_gate = {
    "shared_prefix_tokens": count_stable_prefix_tokens(trace),
    "full_blocks": shared_prefix_tokens // cache_block_size,
    "producer_finished_before_consumer": first_request_done_at < second_request_start_at,
    "blocks_survive": evicted_reusable_blocks == 0,
    "win": p95_ttft_with_cache < p95_ttft_without_cache,
}

1. Replay timing: run back-to-back, simultaneous, and delayed repeated-prefix traffic because scheduler timing changes when blocks become reusable.
2. Vary block size: smaller blocks can expose more reusable prefixes, while larger blocks can improve kernel efficiency but reduce match granularity.
3. Separate phases: report saved prefill tokens and TTFT, then keep decode ITL as a separate guardrail because prefix reuse does not shorten generated-token work.
4. Stress eviction: repeat the test with long outputs and high concurrency so the cache policy proves useful under memory pressure, not only in an empty worker.

Policy Drill: Give Valuable Cache Blocks A Retention Budget

KV reuse is only useful if the blocks survive long enough to be hit. TensorRT-LLM's KV cache docs describe a block pool with cross-request reuse, prioritized LRU eviction, optional host-memory offload, request retention policies for prompt token ranges, and a separate priority for decoded-token blocks. Treat those controls as product policy: a shared system prompt, expensive retrieved context, or active multi-turn session may deserve higher retention than one-off generated tails.

cache_retention_plan = {
    "system_prompt": {"priority": 90, "duration_ms": 10 * 60 * 1000},
    "retrieved_context": {"priority": 70, "duration_ms": 2 * 60 * 1000},
    "generated_tail": {"priority": 35, "duration_ms": 30 * 1000},
    "offload_min_priority": 50,
}

1. Protect what repeats: assign higher retention to stable prefixes that many future requests can reuse, not to every token equally.
2. Expire deliberately: duration limits prevent stale prompt templates, adapters, or tenant-specific contexts from occupying cache after their value is gone.
3. Charge offload latency: host-memory offload can preserve reuse, but reused blocks must be copied back before they help TTFT.
4. Watch eviction counters: compare prefix-hit rate with evicted blocks by priority so a cache policy failure is visible before users report slower first tokens.

Implementation Drill: Quantize KV Cache As A Serving Profile

KV-cache quantization is different from weight quantization. Weight quantization shrinks the resident model; KV-cache quantization shrinks the per-token state that grows with active context and batch size. The KIVI paper studies this bottleneck and proposes asymmetric low-bit cache quantization, while current Hugging Face and vLLM docs expose practical quantized-cache paths for longer contexts or larger batches. Treat the choice as a serving profile that needs its own quality and latency gate, not as a harmless flag.

kv_quant_gate = {
    "baseline": "fp16_or_bf16_kv_cache",
    "candidate": "quantized_kv_cache",
    "expected_win": ["more_active_tokens", "lower_kv_memory", "better_decode_capacity"],
    "quality_slices": ["long_context_recall", "code_exactness", "rag_grounding"],
    "latency_slices": ["ttft", "itl", "cache_dequant_overhead"],
}

1. Separate the ledgers: report model weight memory, KV-cache memory, activation/workspace headroom, and cache dtype independently.
2. Benchmark long context: cache quantization may matter little for short prompts but decide whether long prompts or large batches fit at all.
3. Measure quality by task: run exactness-heavy, retrieval-heavy, and long-context cases because small attention errors can show up unevenly.
4. Record the implementation: note cache dtype, scaling/calibration method, residual length if used, backend, model revision, and rollback profile.

Observability Drill: Separate Cache Correctness From Cache Value

A KV cache rollout has two different questions. Correctness asks whether reused state is valid for the same model, tokenizer, adapter, multimodal inputs, and security namespace. Value asks whether the retained blocks actually reduce TTFT, prefill tokens, memory pressure, or GPU work. vLLM's prefix-cache design includes parent hashes, block tokens, extra hash fields, and optional cache salts; TensorRT-LLM exposes reuse, prioritized eviction, offload, salting, and cache events; Hugging Face shows the smaller single-process version by pre-filling a cache for a stable prefix. Treat those as the minimum logging contract for a production cache experiment.

cache_observability = {
    "correctness_keys": [
        "model_revision", "tokenizer_revision", "adapter_id",
        "cache_namespace", "multimodal_ids", "block_hash_algo",
    ],
    "value_metrics": [
        "prefix_hit_tokens", "requested_prefill_tokens", "ttft_ms",
        "gpu_cache_free_blocks", "evictions_by_priority", "offload_reads",
    ],
    "rollback_trigger": "hit_rate_low or p95_ttft_regressed or collision_risk_changed",
}

1. Log the namespace: every cache hit should be explainable from model, tokenizer, adapter, salt, and prompt-template versions.
2. Count tokens, not just hits: a hit on one tiny block is not worth the same as reusing a long shared system prompt.
3. Watch eviction cause: low hit rate can come from unstable prompts, over-tight salting, too little GPU cache, or priority rules that keep low-value blocks.
4. Keep an off switch: cache reuse should be disableable without changing the target model or generation contract when observability shows no value.

Concept Check: Do Not Confuse Kernel IO With Cache Allocation

FlashAttention, PyTorch SDPA backends, and PagedAttention all appear in serving conversations, but they answer different questions. FlashAttention reduces attention memory traffic by tiling exact attention through fast on-chip memory. PyTorch SDPA is the API that may dispatch to different attention kernels depending on dtype, mask, device, and shape. vLLM's PagedAttention work treats the KV cache as blocks so variable-length requests can be admitted and scheduled without one contiguous cache allocation per sequence. A good design note should name which layer is being optimized before claiming a latency or memory win.

serving_bottleneck_map = {
    "slow_long_prefill": "attention kernel and prompt batching",
    "oom_at_high_concurrency": "KV cache budget, block slack, and admission",
    "decode_itl_regression": "KV reads, active streams, scheduler policy",
    "unexpected_math_fallback": "SDPA backend constraints and logging",
}

1. Kernel question: which exact attention backend ran, and does it preserve logits within tolerance against a reference?
2. Cache question: how many KV blocks are allocated, wasted in tails, reused as prefixes, evicted, or offloaded?
3. Scheduler question: which prefill and decode tokens entered each iteration, and did the policy protect p95 TTFT and ITL?
4. Claim narrowly: report whether the improvement came from lower HBM traffic, better block allocation, more cache reuse, or safer admission.

Kernel Drill: Prove The Fast Attention Path Is Still The Same Contract

FlashAttention and PyTorch SDPA can make attention faster by changing the memory schedule or backend dispatch, not by changing the attention equation. PagedAttention changes where cached keys and values live, not which logical tokens a decode step may attend to. Before launching a new kernel, cache layout, or GQA path, run a tiny equivalence test and record the backend context that made the result true.

kernel_launch_note = {
    "math_contract": "same logits as reference within tolerance",
    "backend_context": ["gpu", "dtype", "head_dim", "mask", "seq_lens"],
    "cache_context": ["block_size", "kv_heads", "cache_dtype", "prefix_reuse"],
    "metrics": ["ttft_ms", "itl_ms", "tokens_per_second", "peak_kv_blocks"],
    "rollback": "return to previous backend or cache profile",
}

1. Separate correctness from speed: first compare fused attention with a slow reference on deterministic inputs, then benchmark realistic prefill and decode lengths.
2. Name the backend: log whether SDPA used FlashAttention, memory-efficient attention, or a math fallback instead of assuming one API call means one kernel.
3. Keep logical positions stable: paged cache block ids may move, but causal masks, RoPE positions, and prefix-cache keys must still describe the same token history.
4. Watch phase metrics: an attention kernel can improve prefill throughput while decode remains limited by KV-cache reads, batching policy, or admission control.

Sources: FlashAttention, PyTorch scaled dot-product attention, PagedAttention paper, vLLM automatic prefix caching design, vLLM prefix-cache hash collision advisory, vLLM quantized KV cache docs, TensorRT-LLM KV cache system, Hugging Face cache strategies, KIVI KV-cache quantization paper.

Speculative decoding attacks the serial decode bottleneck by letting a cheaper drafter propose several tokens, then asking the target model to verify those candidates in parallel. The original method is designed to preserve the target model's output distribution, but production speedup depends on a practical ratio: draft cost, verification cost, accepted tokens per step, and any extra memory needed for draft-model or multi-head speculation state.

speculation_gate = (
    target_decode_is_memory_bound
    and draft_ms_per_token * draft_tokens < saved_target_decode_ms
    and accepted_tokens_per_verify >= 2
    and quality_regression_rate <= launch_threshold
    and gpu_headroom_bytes > speculation_workspace_bytes
)

1. Measure acceptance: log proposed tokens, accepted tokens, fallback tokens, and per-request latency instead of reporting only aggregate tokens per second.
2. Pick the drafter by workload: a small model, n-gram matcher, suffix cache, or learned draft heads can win on different prompt domains and traffic levels.
3. Protect tail latency: speculation can add work when the drafter is inaccurate or the server is already saturated, so compare p95 ITL and GPU utilization against a non-speculative profile.
4. Keep rollback simple: launch speculation as a decoding profile that can be disabled without changing the target model, tokenizer, or eval contract.

Benchmark Drill: Plot Acceptance Before Claiming A Speedup

A speculative profile is only useful if accepted draft tokens reduce target-model decode steps more than the extra draft and verification work costs. Hugging Face's assisted decoding docs emphasize that an assistant model drafts candidate tokens and the main model verifies them in one pass; vLLM's production docs expose several speculation methods, including draft-model, n-gram, suffix, and learned speculators. Benchmark each method as a separate decoding contract, because the best choice depends on prompt repetition, tokenizer compatibility, model alignment, and live GPU saturation.

speculation_report = {
    "method": "draft_model | ngram | suffix | learned_heads",
    "draft_tokens_per_step": sweep([1, 2, 4, 8]),
    "acceptance_rate": accepted_tokens / proposed_tokens,
    "effective_decode_steps": target_forwards / output_tokens,
    "tail_check": p95_itl_speculative <= p95_itl_baseline,
}

1. Sweep lookahead: increasing draft tokens can help when acceptance stays high, but it wastes work when rejections begin early.
2. Check tokenizer assumptions: same-family assistant models are simpler; cross-tokenizer assisted generation needs an explicit retokenization path.
3. Separate traffic slices: repetitive code completion, templated RAG answers, and open-ended chat may have very different acceptance curves.
4. Watch saturation: a draft model that helps an idle target can hurt a busy server if it competes for the same memory, kernels, or batch slots.

Implementation Drill: Turn Acceptance Into A Latency Budget

Before enabling speculation in a shared serving pool, estimate the breakeven point in decode steps rather than assuming every accepted token is free. The original speculative decoding paper motivates the speedup by reducing serial target-model calls, while current vLLM docs caution that the method is most useful for medium-to-low QPS, memory-bound decode workloads. A rollout gate should therefore combine acceptance rate with the cost of the drafter, verification pass, and any lost batching efficiency.

def speculation_wins(target_ms, draft_ms, verify_ms, draft_tokens, accepted_tokens):
    if accepted_tokens == 0:
        return False
    baseline_ms = accepted_tokens * target_ms
    speculative_ms = draft_tokens * draft_ms + verify_ms
    return speculative_ms < baseline_ms

rollout_gate = (
    speculation_wins(target_decode_ms, draft_decode_ms, target_verify_ms,
                     proposed_tokens, accepted_tokens)
    and qps_bucket in ["low", "medium"]
    and p95_itl_speculative <= p95_itl_baseline
)

1. Count accepted tokens per verify pass: this is the practical unit of serial decode work saved.
2. Charge the drafter honestly: include draft-model latency, tokenizer conversion if any, memory pressure, and queue slots.
3. Fail closed under load: disable or shrink speculation when the target is compute-saturated, acceptance drops, or tail ITL regresses.

Reliability Drill: Add A Speculation Kill Switch

Speculative decoding should be a runtime profile, not a permanent assumption baked into the service. The original algorithm preserves the target distribution when the accept/reject step is implemented correctly, but production frameworks still warn that the speedup is workload-dependent. Use a short-window controller that can shrink the draft length or fall back to ordinary decoding when acceptance, tail latency, or memory headroom moves outside the launch envelope.

def choose_decode_profile(metrics):
    if metrics.gpu_cache_free_blocks < metrics.min_free_blocks:
        return {"speculative": False}
    if metrics.accepted_tokens_per_verify < 1.5:
        return {"speculative": False}
    if metrics.p95_itl_ms > metrics.baseline_p95_itl_ms * 1.05:
        return {"speculative": False}
    if metrics.queue_depth > metrics.high_qps_queue_depth:
        return {"speculative": False}
    return {"speculative": True, "draft_tokens": metrics.best_recent_draft_tokens}

1. Window the metrics: evaluate acceptance and ITL over recent requests so one hard prompt does not disable the whole fleet.
2. Fall back by profile: keep the target model, tokenizer, and sampling contract unchanged; only disable the draft path.
3. Record the reason: log whether the fallback came from low acceptance, queue pressure, memory pressure, or tail-latency regression.

Config Drill: Version The Speculation Contract

Treat a speculative-decoding rollout as a versioned serving contract, not a hidden engine flag. vLLM's current configuration centers on a speculative_config object with fields such as method, model, num_speculative_tokens, draft tensor-parallel size, and rejection-sampling mode. The original speculative-sampling paper preserves the target distribution through the verifier and modified rejection sampling, so the production contract should record both the algorithmic mode and the sampling settings that were validated.

speculation_contract = {
    "target_model": "repo/name@revision",
    "sampling": {"temperature": 0.7, "top_p": 0.95},
    "speculative_config": {
        "method": "draft_model | eagle3 | mtp | ngram | suffix",
        "model": "draft-or-speculator@revision",
        "num_speculative_tokens": 4,
        "draft_tensor_parallel_size": 1,
        "rejection_sample_method": "strict",
    },
    "validated_metrics": ["acceptance_rate", "p95_itl", "quality_delta"],
}

1. Pin revisions: record target, draft, tokenizer, adapter, and serving-engine versions so acceptance changes can be traced.
2. Keep sampling outside config: temperature and top-p are request sampling parameters, not speculative-config fields, but they still affect acceptance and must be part of the benchmark contract.
3. Use compatibility checks: verify the installed engine supports the selected method and parallelism mode before comparing latency.
4. Report the fallback: include the non-speculative profile that preserves the same target model and sampling settings.

Control Drill: Let Lookahead Shrink When Confidence Drops

A fixed number of draft tokens is easy to benchmark, but it can waste work on prompts where the assistant becomes uncertain after one or two tokens. Hugging Face's dynamic speculation notes stop drafting when assistant confidence falls below a threshold, while vLLM lists dynamic speculative decoding alongside fixed draft-model, n-gram, suffix, MTP, and EAGLE-style methods. Treat draft length as a controller output, not a constant, and log why each speculative step stopped.

def choose_draft_budget(recent, config):
    if recent.queue_depth > config.queue_guard:
        return 0
    if recent.accepted_tokens_per_verify < config.min_accept:
        return max(1, recent.draft_tokens - 1)
    if recent.assistant_confidence_p10 < config.confidence_floor:
        return max(1, recent.draft_tokens - 1)
    if recent.p95_itl_ms <= recent.baseline_p95_itl_ms:
        return min(config.max_draft_tokens, recent.draft_tokens + 1)
    return 0

1. Bound both sides: set a maximum draft length and a confidence floor so the assistant can stop early instead of forcing low-quality proposals.
2. Track stop reasons: count verifier rejection, confidence stop, max-lookahead stop, queue-pressure stop, and tail-latency fallback separately.
3. Compare schedules: benchmark constant, heuristic, and dynamic lookahead on the same prompt slices before changing the draft model.

Weight quantization and KV-cache quantization solve different serving problems. Lower-precision weights reduce model-loading memory and can make a larger model fit on the target device. KV-cache quantization reduces the persistent memory that grows with active sequences, prompt length, generated tokens, layer count, and KV heads. In a long-context or high-concurrency service, cache dtype can matter more than weight dtype after the model is already resident.

quantization_decision = {
    "weights": "use when model weights do not fit or weight bandwidth dominates",
    "kv_cache": "use when active tokens or long contexts cap concurrency",
    "acceptance": "compare quality, TTFT, ITL, memory headroom, and fallback path",
}

1. Start with a memory trace: split resident weight memory from KV-cache growth under realistic prompt and output lengths.
2. Keep calibration explicit: cache scales, static versus dynamic quantization, and hardware support are runtime choices, not just model-card labels.
3. Test the failure shape: long prompts, retrieval-heavy contexts, and multi-turn sessions are more likely to expose cache-precision regressions than short chat smoke tests.
4. Ship with rollback: treat quantized weights, FP8 KV cache, and default cache dtype as separate launch profiles with comparable eval and latency reports.

Rollout Drill: Separate Weight Dtype From Cache Dtype

Treat quantization as a matrix, not a single switch. Hugging Face's bitsandbytes path changes how model weights are loaded, while vLLM and TensorRT-LLM expose KV-cache dtype controls such as FP8 cache and, on supported stacks, lower-precision cache formats. vLLM also distinguishes uncalibrated FP8 cache, warmup-scale estimation, and dataset calibration through llm-compressor, so the serving profile should record the scale source alongside the dtype. A useful rollout compares each profile on the same prompt buckets before changing max context or concurrency.

quantization_matrix = [
    {"weights": "bf16", "kv_cache": "bf16", "calibration": "none"},
    {"weights": "int8", "kv_cache": "bf16", "calibration": "weight loader"},
    {"weights": "int4", "kv_cache": "bf16", "calibration": "weight loader"},
    {"weights": "bf16", "kv_cache": "fp8", "calibration": "dataset scales"},
    {"weights": "fp8", "kv_cache": "fp8", "calibration": "engine-specific"},
]

for profile in quantization_matrix:
    run_eval(profile, buckets=["short_chat", "long_rag", "multi_turn"])
    compare(profile, metrics=["quality", "ttft", "itl", "oom_rate", "cost"])

1. Record support: note model family, GPU generation, serving engine, and whether the chosen cache dtype is native or requires offline quantization.
2. Calibrate on the workload: use representative long-context and retrieval prompts when cache scales are estimated from data.
3. Compare phase metrics: weight quantization may improve fit or load memory, while KV-cache quantization should show up in active-token capacity, OOM rate, and long-context TTFT/ITL.
4. Keep rollback orthogonal: be able to disable cache quantization without changing the weight checkpoint, tokenizer, sampling contract, or eval threshold.

Smoke Test: Make The Cache Backend Observable

A cache-quantization experiment should leave an audit trail before it changes service limits. Hugging Face Transformers exposes quantized KV cache through cache_implementation="quantized" with backend choices such as quanto and hqq, while the KIVI paper motivates why keys and values may need different quantization treatment. Keep those details in the benchmark output so a future regression can be traced to backend, bit width, residual window, or workload rather than a vague "quantization changed" note.

cache_profile = {
    "cache_implementation": "quantized",
    "backend": "quanto",
    "nbits": 4,
    "residual_length": 128,
    "axis_key": 0,
    "axis_value": 0,
}

report = run_generation_probe(
    model_id=model_id,
    cache_profile=cache_profile,
    prompts=load_prompt_bucket("long_rag_and_exact_copy"),
    metrics=["peak_memory", "ttft", "itl", "tokens_per_second", "quality_delta"],
)

1. Log the backend: record cache implementation, backend, bit width, axis choices, residual length, model revision, and tokenizer revision with every benchmark run.
2. Compare one change: hold weights, sampling, prompt bucket, and max tokens fixed while changing only the cache profile.
3. Include rollback evidence: save the baseline memory and latency counters beside the candidate so disabling cache quantization is a known path, not a guess.

Validation Drill: Test The Cache Where Attention Is Fragile

KV-cache quantization changes the stored keys and values that every later decode step attends over, so validate it on prompts where small attention-score changes matter. KIVI motivates asymmetric cache quantization by showing that key and value caches have different distribution patterns, while Hugging Face's quantized cache keeps a recent residual cache in full precision before quantizing older states. That means a launch test should include long-range retrieval, multi-turn references, exact copying, and reasoning prompts, not only short conversational examples where the cache is small.

kv_quant_eval = {
    "baseline": {"kv_cache": "bf16"},
    "candidate": {"kv_cache": "quantized", "residual_length": 128},
    "buckets": ["short_chat", "long_rag", "needle_retrieval", "exact_copy"],
    "checks": ["answer_match", "citation_span", "loop_rate", "ttft", "itl", "oom_rate"],
}

1. Stress old tokens: include prompts where the needed evidence appears far before the generated answer, because those K/V states are most likely to be quantized.
2. Watch copy fidelity: names, numbers, code, and citations expose small attention errors faster than loose chat preferences.
3. Separate residual length: sweep the unquantized recent-cache window independently from bit width so memory savings and quality loss are attributable.
4. Compare cache and weight changes: keep the weight checkpoint fixed while testing cache dtype, then test combined weight-plus-cache quantization as a separate profile.

A production LLM endpoint needs traces and metrics that preserve the shape of the request. vLLM's metrics separate running and waiting requests, GPU cache usage, prompt-token throughput, generation-token throughput, and prefix-cache hit rate. OpenTelemetry's GenAI semantic conventions give client and gateway traces common fields for operation, model, provider, and token usage. TensorRT-LLM exposes iteration-level serving stats such as queued requests, active requests, batch sizes, memory, KV-cache state, in-flight batching, and speculative decoding.

serving_trace = {
    "request": ["model", "tenant", "prompt_tokens", "max_new_tokens"],
    "queue": ["waiting_requests", "queue_ms"],
    "prefill": ["prompt_tokens_per_s", "ttft_ms", "prefix_cache_hit_rate"],
    "decode": ["generation_tokens_per_s", "itl_ms", "accepted_spec_tokens"],
    "memory": ["gpu_cache_usage", "kv_blocks_free", "oom_or_eviction_count"],
}

1. High TTFT: inspect queue time, prompt length, chunked-prefill budget, prefix-cache hit rate, and prefill worker saturation before blaming decode kernels.
2. High ITL: inspect active streams, decode batch size, KV-cache bandwidth, speculative acceptance, and GPU cache pressure.
3. Sudden OOM: compare promised tokens, real output lengths, cache block slack, adapter count, CUDA graph memory, and rollout changes.
4. Cost drift: group input tokens, output tokens, cached-prefix reuse, model revision, and decoding profile so product changes are visible in token spend.

Dashboard Drill: Split User SLOs From Engine Counters

A useful serving dashboard has two layers. The user layer tracks end-to-end latency, TTFT, time-to-second-token, ITL or TPOT, error rate, and stream completion. The engine layer tracks waiting requests, running requests, prompt throughput, output throughput, cache usage, prefix-cache hits, and speculative acceptance. vLLM exposes production metrics for request and token phases; NVIDIA's NIM and GenAI-Perf docs define TTFT, ITL, output-token throughput, and request throughput as benchmark-facing metrics. Keep both views because an SLO page tells you what users felt, while engine counters explain which phase caused it.

dashboard = {
    "user_slo": ["p50_ttft", "p95_ttft", "p95_itl", "p99_e2e", "stream_errors"],
    "engine": ["waiting_requests", "running_requests", "gpu_cache_usage",
               "prompt_tokens_per_s", "output_tokens_per_s", "prefix_cache_hit_rate"],
    "slices": ["model_revision", "adapter_id", "decode_profile", "prompt_bucket"],
}

1. Alert on symptoms: page on user-facing TTFT, ITL, error rate, and saturation, not on every low-level counter twitch.
2. Debug by phase: if TTFT rises with queue depth, look at admission and prefill; if ITL rises with running requests, look at decode batch, KV bandwidth, and cache pressure.
3. Slice before changing knobs: separate short chat, long RAG prompts, tool-heavy requests, LoRA adapters, and speculative profiles before tuning a global scheduler setting.
4. Compare benchmark and prod: a benchmark can report clean TTFT and ITL, but production traces must add gateway, network, retry, and streaming-client behavior.

Implementation Drill: Reconstruct Streaming Latency From Events

Benchmark tools often report TTFT, ITL, TPOT, throughput, and end-to-end latency directly, but production traces should be able to recompute them from event timestamps. vLLM documents TTFT and time-per-output-token metrics, TensorRT-LLM separates TPOT from ITL, and Hugging Face TGI exposes Prometheus metrics for server monitoring. Store the raw stream events first, then derive dashboard numbers so you can audit definitions when a benchmark and production graph disagree.

events = {
    "received_at": t0,
    "prefill_done_at": t_prefill,
    "token_sent_at": [t1, t2, t3, t4],
    "completed_at": t_done,
}

ttft = events["token_sent_at"][0] - events["received_at"]
itl = [b - a for a, b in pairwise(events["token_sent_at"])]
tpot = mean(itl)
e2e = events["completed_at"] - events["received_at"]

1. Name the boundary: decide whether TTFT starts at client send, gateway receive, engine receive, or scheduler enqueue, then keep that label visible.
2. Keep token events: aggregate histograms are useful, but sampled per-request traces explain bursts, stalls, retries, and stream cancellations.
3. Separate empty outputs: requests that fail, stop before emitting a token, or stream tool metadata need their own counters instead of fake TTFT or ITL values.
4. Compare definitions: when two tools disagree, check whether they include tokenization, queueing, network flush, and the first generated token in the same way.

Cache Drill: Explain TTFT With Reuse, Not Averages

Prefix-cache reports should connect cache hits to the phase they actually save. vLLM's automatic prefix caching and TensorRT-LLM's KV cache reuse both skip redundant prefill work when requests begin with matching prompt blocks, so the first expected win is lower TTFT for repeated prefixes. Hugging Face cache strategies describe per-request decode caches as a generation optimization, which means a prefix hit should not be credited for faster output-token work unless decode pressure also changed. Keep cache-hit metrics sliced by prompt template, tenant or namespace, adapter, and model revision so eviction or salting decisions are visible.

cache_latency_report = {
    "slice": ["template_id", "tenant_policy", "model_revision", "adapter_id"],
    "reuse": ["prefix_cache_hit_rate", "reused_prompt_tokens", "evicted_blocks"],
    "latency": ["queue_ms", "prefill_ms", "ttft_ms", "itl_ms"],
    "memory": ["gpu_cache_usage", "kv_blocks_free"],
}

1. Compare matched slices: measure cached and uncached requests with the same prompt bucket and output budget before claiming a global latency win.
2. Charge evictions: a high hit rate can still regress tails if useful blocks are evicted during prompt spikes or adapter-heavy traffic.
3. Keep decode separate: prefix reuse reduces repeated prefill compute; ITL still depends on active streams, KV bandwidth, and decode batch scheduling.
4. Audit namespace changes: model revisions, tokenizer changes, LoRA adapters, and tenant salts can intentionally split reuse and should appear in the report.

JSON mode, grammar guidance, and structured outputs are production controls, not just prompt style. OpenAI documents schema-backed structured outputs with strict JSON Schema, vLLM exposes structured output constraints for choices, regexes, JSON schemas, grammars, and structural tags, and Hugging Face TGI supports JSON and regex grammar guidance. Treat the schema, refusal path, max-token behavior, and backend support as part of the endpoint contract before downstream code parses the response.

structured_output_contract = {
    "schema_name": "citation_answer_v3",
    "schema_revision": "2026-07-02",
    "strict": True,
    "fields": ["answer", "citations", "confidence"],
    "edge_cases": ["refusal", "truncated_output", "schema_unsupported"],
    "metrics": ["parse_success_rate", "schema_error_rate", "ttft_ms", "itl_ms"],
}

1. Version the schema: include schema name, revision, model revision, and serving backend in traces so parser failures can be tied to a contract change.
2. Test edge cases: refusals, content filters, max-token truncation, unsupported schema features, and stream cancellation need separate counters from ordinary parse failures.
3. Benchmark the constraint: constrained decoding can change latency and token choices, so compare TTFT, ITL, completion length, and task quality against the unconstrained baseline.
4. Parse after validation: downstream services should only consume typed objects after the response has matched the promised schema or entered a documented fallback path.

Sources: OpenAI structured outputs guide, vLLM structured outputs, Hugging Face TGI guidance.

A serving launch is not just a faster endpoint. It needs regression prompts, load tests, OOM behavior, streaming behavior, safety gates, cost monitoring, and a rollback path to the previous model or decoding configuration.

minimum_launch_checks = [
    "golden prompts unchanged or explained",
    "p50/p95 latency under expected concurrency",
    "OOM returns controlled errors, not worker death loops",
    "streaming emits first token within target",
    "rollback command tested before launch",
]

Sources: PagedAttention paper, vLLM documentation, vLLM paged attention design, vLLM optimization and chunked prefill, vLLM LoRA adapters, Hugging Face TGI overview, Hugging Face TGI LoRA, Hugging Face TGI serving engine notes, vLLM quantization, vLLM quantized KV cache, vLLM performance and chunked prefill tuning, vLLM prefix caching design, vLLM automatic prefix caching, SGLang documentation, SGLang hyperparameter tuning, LMSYS SGLang and RadixAttention blog, SGLang / RadixAttention paper, OpenAI prompt caching, vLLM metrics, vLLM production metrics, vLLM cache salting design discussion, vLLM disaggregated prefilling, Ray Serve prefill/decode disaggregation, Ray Serve multi-LoRA deployment, TensorRT-LLM documentation, TensorRT-LLM overview, TensorRT-LLM quantization, TensorRT-LLM KV cache system, TensorRT-LLM KV cache reuse, TensorRT-LLM metrics collector, TensorRT-LLM KV cache configuration, TensorRT-LLM benchmark metrics, TensorRT-LLM disaggregated serving, TensorRT-LLM paged KV cache notes, NVIDIA NIM LLM benchmarking metrics, NVIDIA GenAI-Perf, Hugging Face TGI metrics, Hugging Face cache strategies, Hugging Face KV cache quantization, OpenTelemetry GenAI semantic conventions, OpenAI structured outputs guide, vLLM structured outputs, Hugging Face TGI guidance, Hugging Face bitsandbytes quantization, KIVI KV-cache quantization, Accelerating Large Language Model Decoding with Speculative Sampling, Fast Inference from Transformers via Speculative Decoding, vLLM speculative decoding, vLLM EAGLE speculative decoding example, Hugging Face assisted decoding, Hugging Face dynamic speculation lookahead, Decoding Speculative Decoding.