Module 15
A practical order for reading the papers and docs that support the rest of the course.
Reading Method
Architecture
Learn self-attention, multi-head attention, residual paths, layer normalization, position encodings, warmup schedules, and why parallel sequence modeling displaced recurrent training for many tasks.
Implementation check: write scaled dot-product attention, then explain every dimension in Q, K, V, scores, weights, and output.
Learn why positional information can be applied by rotating Q/K pairs, giving attention scores a relative-position structure without adding position vectors to values.
Implementation check: apply RoPE to Q/K and test that dot products depend on relative offsets.
Inference
Learn why exact attention can be faster by changing memory access patterns rather than approximating attention scores.
Implementation check: compare naive attention memory use with PyTorch's scaled dot-product attention on increasing sequence lengths.
Read this as a hardware-aware sequel: the attention math stays exact, while Hopper-specific asynchrony, warp specialization, Tensor Memory Accelerator movement, and FP8 paths change the kernel bottleneck.
Implementation check: write a short compatibility note that separates algorithmic invariants from hardware requirements such as H100/H800-class GPUs, CUDA support, dtype, head dimension, dropout, and variable-length behavior.
Learn the production-facing API surface for attention: masks, dropout, causal mode, grouped-query attention support, and backend selection behavior.
Implementation check: replace a handwritten attention function with F.scaled_dot_product_attention and keep tests identical.
Read GQA after multi-query attention: MQA shares one K/V head to reduce incremental-decoding memory bandwidth, while GQA keeps an intermediate number of K/V heads to recover more quality while preserving much of the serving win.
Implementation check: compute KV cache bytes for MHA, MQA, and GQA, then write the shape test that repeats compact K/V heads to match query heads only inside attention.
Learn why KV cache memory management determines serving throughput for variable-length requests and how block-based cache paging reduces waste.
Implementation check: estimate KV cache bytes and sketch a continuous batching admission rule.
Read this after PagedAttention: prefill and decode stress different resources, so a serving system can scale and schedule them separately when TTFT and time-per-output-token SLOs conflict. Pair it with vLLM disaggregated prefilling and Ray Serve prefill/decode disaggregation to see the production version of the idea.
Implementation check: replay one mixed workload and report TTFT, TPOT/ITL, KV-transfer time, and the fallback condition where one pooled deployment is still better.
Read speculative decoding as a latency method, not a new model objective: a cheaper drafter proposes several tokens, the target model verifies them in parallel, and the acceptance rule preserves the target distribution when its assumptions hold. Pair the paper with Hugging Face assisted decoding and vLLM speculative decoding to see how the idea turns into serving knobs such as assistant models, n-gram proposals, speculation length, and rollback profiles.
Implementation check: benchmark baseline decoding against one speculative profile while holding the target model, prompt set, sampling parameters, max tokens, and output-quality checks constant.
Adaptation
Learn how freezing base weights and training low-rank update matrices cuts trainable parameters and optimizer memory for adaptation.
Implementation check: implement LoRA for a linear layer and count trainable parameters as rank changes.
Learn how preference pairs can train a language model directly without a separate reward-model-plus-RL loop.
Implementation check: write the shape contract for chosen/rejected log probabilities and explain the role of the reference model.
Retrieval
Learn the distinction between parametric memory in model weights and non-parametric memory in a retrievable index. For applied LLM systems, RAG is often a provenance, freshness, and evaluation design problem as much as a modeling problem.
Implementation check: build a tiny retrieval eval with query, retrieved passages, answer, citation, and groundedness fields.
Evaluation
Learn the core eval loop: define the production behavior, collect representative cases, choose metrics that match the failure mode, compare candidates, and keep evaluating as the app changes. Note that OpenAI's hosted Evals platform is being deprecated, so treat the guide as design advice and keep your durable cases portable.
Implementation check: write five eval rows with case id, expected behavior, scorer, slice, severity, and baseline/candidate outputs.
Learn why model evaluation should report scenarios and multiple metrics together. Accuracy alone can hide tradeoffs in calibration, robustness, fairness, bias, toxicity, and efficiency.
Implementation check: make a slice report that shows quality, latency, cost, and safety deltas separately before computing any aggregate score.
Learn to split a RAG failure into retrieval quality, context usefulness, answer relevance, and faithfulness. This keeps a retrieval miss from being hidden by a fluent generator, and keeps a generator hallucination from being blamed on the index.
Implementation check: score one RAG case with retrieved source ids, expected source ids, answer claims, citation support, and a faithfulness verdict.
Agents
Learn why reasoning and acting are interleaved in tool-using systems: the model chooses actions, observes external results, updates its plan, and keeps a trajectory that humans can inspect.
Implementation check: define a trace schema with thought or decision type, tool name, validated arguments, observation, retry reason, and final answer.
Learn the tool-use contract as a modeling target: decide when to call an API, what arguments to pass, and how the returned value should condition later tokens.
Implementation check: label five examples where a tool call is useful, then write the acceptance test that decides whether the tool result improved next-token likelihood or final-task accuracy.
Read agent tool use as a security boundary: untrusted prompts, retrieved text, tool outputs, and connector metadata should not become authority to read, write, send, delete, or publish.
Implementation check: design a tool-call policy that records the data source, required permission, user approval state, and a denial reason for prompt-injection attempts.
Learn why prompt injection is an application risk, not just a bad prompt: external content can try to override instructions, exfiltrate secrets, or trigger unauthorized tools.
Implementation check: add three regression cases where a retrieved page or tool result contains malicious instructions, then verify the agent treats that text as data only.