Theory
Explain probability, cross entropy, attention, tokenization, optimization, and scaling in plain language.
North Star
Move From Equations To Working LLM Components
PyTorch
Implement the core building blocks with clear shapes, masks, losses, schedules, and edge cases.
Systems
Reason about context length, KV cache memory, batching, serving latency, fine-tuning, RAG, and evaluation.
Study Rhythm
Use The Explain, Implement, Stress-Test Loop
- Explain: state the concept in 30 seconds, including why it exists.
- Implement: write the smallest correct function or module from memory.
- Stress-test: check shape, dtype, mask, empty-row, long-context, and numerical-stability cases.
- Connect: map the concept to training, inference, and product behavior.
- Review: answer one interview-style prompt without looking at notes.
Curriculum
Module Tour
The course is LLM-first: start with ML math and PyTorch, then build transformers, decoding, fine-tuning, retrieval, agents, evaluation, and production serving. The attached PDFs informed the drill topics, especially the hand-written coding questions.
Orientation And Tooling
Set up Python, PyTorch, notebooks, Git, experiment logs, and a reproducible local study loop.
Lab: run a tensor shape notebook and save a one-page setup note.
ML Math For LLMs
Linear models, logistic regression, softmax regression, KL, cross entropy, likelihood, and gradients.
Lab: derive softmax cross entropy and implement it with stable logits.
PyTorch Foundations
Tensors, broadcasting, autograd, modules, losses, normalization, optimizers, and training loops.
Lab: implement stable softmax, CE, layer norm, RMSNorm, AdamW, and clipping.
Tokenization And Embeddings
BPE intuition, vocabulary tradeoffs, embedding lookup, tied embeddings, and sequence length budgeting.
Lab: write a tiny BPE merge trainer and tokenizer for toy text.
Transformer Internals
Scaled dot-product attention, masks, MHA, GQA, MQA, FFN, SwiGLU, residual paths, and normalization order.
Lab: implement single-head attention, expand it to multi-head attention, then test a packed-document attention mask.
Position And Context
Learned positions, sinusoidal encodings, RoPE, ALiBi-style bias, context extension, and extrapolation risks.
Lab: apply RoPE to Q/K and test shape preservation.
Pretraining Objective
Causal language modeling, teacher forcing, next-token loss, masking, perplexity, and data mixing.
Lab: train a tiny character LM and inspect failure modes.
Decoding And Sampling
Greedy, beam search, temperature, top-k, top-p, repetition penalty, no-repeat n-gram, stop conditions, and speculative decoding.
Lab: implement sampling helpers, then sketch when an assistant model can speed up target-model decoding.
KV Cache And Serving Math
Prefill, decode, cache layout, GQA/MQA memory savings, context windows, batching, and throughput/latency tradeoffs.
Lab: estimate KV memory for a model and batch under different context lengths and KV-head counts.
Fine-Tuning And Alignment
SFT, instruction tuning, preference tuning, DPO-style objectives, LoRA, adapters, and eval gates.
Lab: implement a LoRA linear layer and count trainable parameters.
RAG And Tool Use
Chunking, embeddings, retrieval, reranking, grounding, tool schemas, memory, and failure analysis.
Lab: build a tiny retrieval eval with exact-match and groundedness notes.
LLM Agents
Planning, tool calls, state, retries, guardrails, multi-turn memory, and when not to use an agent loop.
Lab: design an agent trace schema and write three failure-case tests.
Evaluation
Unit tests, golden sets, pairwise eval, rubrics, judges, hallucination checks, safety, and regression gates.
Lab: write a small eval suite for instruction following and retrieval grounding.
Production Serving
Continuous batching, PagedAttention concepts, quantization, autoscaling, observability, cost, and rollback.
Lab: choose a serving plan for an interactive assistant with cost and p95 latency targets.
Interview Drill Pack
Explain-first coding tasks for ML math, PyTorch components, transformers, decoding, KV cache, LoRA, and BPE.
Lab: solve five prompts in 45 minutes and review edge cases.
Research Reading Map
Read the primary LLM papers and official docs in a practical order: architecture, position, efficient attention, serving, adaptation, preferences, RAG, and agents.
Lab: summarize one source into "what problem, what method, what tradeoff, what implementation test."
Implementation Spine
What You Should Be Able To Hand-Write
Numerics
Stable softmax, cross entropy, BCE with logits, clipping, log-sum-exp, label smoothing, and safe masking.
Blocks
LayerNorm, RMSNorm, residual links, attention, MHA, GQA, MQA, FFN, SwiGLU, RoPE, and LoRA.
Loops
AdamW, warmup cosine schedules, gradient accumulation, early stopping, EMA, eval loops, and checkpointing.
Systems Lens
Connect Model Internals To Runtime Behavior
Inference
Prefill is parallel over prompt tokens; decode is often memory-bandwidth bound one token at a time. KV cache turns repeated attention over old tokens into reads from cached keys and values.
Product Quality
Sampling settings, context packing, retrieval precision, tool schemas, and evaluation gates directly shape user-visible correctness, latency, cost, and trust.
Practice
Explain Before You Code
Prompt 1: Stable Softmax
Implement stable softmax for an arbitrary dimension. Explain why subtracting the max does not change the probabilities.
Prompt 2: Causal Attention
Implement scaled dot-product attention with both padding and causal masks. Explain the shape of every intermediate tensor.
Prompt 3: Top-p Sampling
Implement nucleus sampling and explain why sorting positions must be mapped back to original token IDs.
Prompt 4: KV Cache
Sketch incremental decoding pseudocode. Explain how cache size scales with layers, batch, heads, sequence length, and head dimension.
Research Backbone
Primary Sources To Read Alongside The Lessons
The course uses the PDFs as topic/index references, then fills the material from public papers and official documentation. Start with the Transformer paper, then connect each later source to a concrete PyTorch or serving exercise.
Architecture And Training
Inference And Retrieval
FlashAttention, PyTorch SDPA, GQA, vLLM / PagedAttention, RAG, and HELM.