Fine-Tuning, RAG, And Agents

Supervised fine-tuning changes model weights. It is useful for response format, domain style, tool-call patterns, and task behavior that repeats across many requests. It is not the best first choice for frequently changing facts.

class LoRALinear(torch.nn.Module):
    def __init__(self, base, rank=8, alpha=16):
        super().__init__()
        self.base = base
        self.base.weight.requires_grad_(False)
        self.a = torch.nn.Linear(base.in_features, rank, bias=False)
        self.b = torch.nn.Linear(rank, base.out_features, bias=False)
        self.scale = alpha / rank

    def forward(self, x):
        return self.base(x) + self.b(self.a(x)) * self.scale

QLoRA keeps the pretrained model frozen and quantized, then backpropagates through it into small LoRA adapter weights. The paper's practical memory recipe combines NF4 quantization, double quantization, and paged optimizers. Treat that as a training configuration, not as permission to update or permanently save the base checkpoint in 4-bit form.

qlora_contract = {
    "base_weights": "frozen 4-bit model used for forward/backward",
    "trainable_weights": "LoRA adapter matrices only",
    "optimizer_state": "adapter parameters, often memory-reduced",
    "merge_for_serving": "reload base at serving dtype, then merge or load adapter",
}

1. Freeze audit: after wrapping the model, count trainable parameters and confirm only adapter modules require gradients.
2. Dtype audit: record compute dtype, quantization type, and double-quantization setting so a later run can reproduce memory and quality.
3. Adapter audit: save adapter config, tokenizer changes, and target module names; a missing module match can silently leave capacity on the table.
4. Serving audit: decide whether production loads base plus adapter, merges adapters into a higher-precision base, or serves multiple adapters on one resident base model.

Memory Drill: Budget What QLoRA Does Not Remove

QLoRA reduces the resident base-weight footprint, but the run still needs memory for activations, LoRA weights, adapter optimizer state, gradients, temporary buffers, and the current batch. The QLoRA paper introduces paged optimizers to smooth optimizer-state spikes, while Hugging Face's PEFT quantization guide shows the practical contract: load the base in 4-bit, prepare it for k-bit training, then attach LoRA adapters as the only trainable weights. A good setup note should record both the quantization recipe and the batch-shape recipe.

qlora_memory_ledger = {
    "frozen_base": "4-bit NF4 + double_quant + compute_dtype",
    "trainable": ["lora_A", "lora_B"],
    "still_scales_with_batch": ["activations", "attention_mask", "sequence_length"],
    "optimizer": "paged_adamw_32bit_or_equivalent",
    "fit_knobs": ["micro_batch", "gradient_accumulation", "checkpointing", "max_seq_len"],
}

1. Start from sequence length: long examples can OOM through activations even when the 4-bit base weights fit comfortably.
2. Record effective batch: separate micro-batch size from gradient accumulation so quality changes are not blamed on quantization alone.
3. Check trainable names: print adapter module names and trainable parameter counts before paying for a long run.
4. Use a smoke batch: run one forward, backward, optimizer step, and save/load cycle before launching full fine-tuning.

Sources: QLoRA paper, Hugging Face PEFT LoRA docs, Hugging Face PEFT quantization guide, Hugging Face Transformers bitsandbytes docs, Hugging Face bitsandbytes optimizer docs.

Classic RLHF usually trains a reward model and then optimizes a policy with reinforcement learning. Direct Preference Optimization reframes preference tuning as a supervised objective over chosen and rejected responses, making the training loop simpler to implement and reason about.

Implementation Drill: Audit The DPO Pair Before Training

A DPO batch should make the preferred completion more likely relative to the rejected completion while comparing that change against a frozen reference policy. In practice, the dataset contract is just prompt, chosen, and rejected, but most bugs come from silent label swaps, chat-template mismatches, missing reference log probabilities, or a beta value that changes the strength of the preference update more than intended.

def dpo_pair_loss(policy_chosen, policy_rejected,
                  ref_chosen, ref_rejected, beta=0.1):
    policy_margin = policy_chosen - policy_rejected
    ref_margin = ref_chosen - ref_rejected
    logits = beta * (policy_margin - ref_margin)
    return -torch.nn.functional.logsigmoid(logits).mean()

dpo_record = {
    "prompt": "Explain KV cache in one sentence.",
    "chosen": "KV cache stores past keys and values so decode can reuse them.",
    "rejected": "KV cache stores gradients for later backpropagation.",
}

1. Validate labels: inspect random pairs and require that chosen is actually preferred under the rubric before any GPU run.
2. Template both sides: apply the same tokenizer and chat template to chosen and rejected completions so length or role formatting does not become the signal.
3. Track margins: log chosen reward, rejected reward, reward accuracy, and loss; a falling loss with worsening held-out preference accuracy is a warning sign.
4. Keep a reference path: decide whether the reference policy is a separate frozen model, precomputed reference log probabilities, or the base model recovered by disabling adapters.

Implementation Drill: Score The Completion, Not The Prompt

DPO compares two completions for the same prompt. The prompt tokens are conditioning context, so their log probabilities should not dominate the chosen-versus-rejected margin. TRL's DPO docs make the dataset contract explicit as prompt, chosen, and rejected, and define the loss over pi(y | x): the completion y given prompt x. In a hand-written trainer, build a completion mask and reduce only over answer tokens after the chat template has been applied.

def sequence_logprob(logits, labels, completion_mask):
    # logits: batch, seq, vocab; labels: batch, seq
    token_logps = torch.log_softmax(logits[:, :-1], dim=-1)
    next_labels = labels[:, 1:]
    next_mask = completion_mask[:, 1:].to(token_logps.dtype)
    gathered = token_logps.gather(-1, next_labels[..., None]).squeeze(-1)
    return (gathered * next_mask).sum(dim=-1)

chosen_logp = sequence_logprob(chosen_logits, chosen_ids, chosen_completion_mask)
rejected_logp = sequence_logprob(rejected_logits, rejected_ids, rejected_completion_mask)
assert chosen_completion_mask.sum() > 0 and rejected_completion_mask.sum() > 0

1. Mask after templating: chat roles, separators, and assistant prefixes can add tokens, so derive the completion mask from tokenized examples, not raw string length.
2. Keep prompts paired: chosen and rejected examples should share the same prompt tokens; a prompt mismatch turns DPO into a formatting comparison.
3. Choose the reduction deliberately: summed log-probability matches the usual sequence likelihood, while length-normalized variants are separate loss choices that need their own eval.

Reasoning Drill: GRPO Needs A Reward Contract

Group Relative Policy Optimization is useful when the training signal can be computed on model-generated completions, such as math accuracy, code tests, format checks, or a calibrated reward function. Unlike DPO, it is an online loop: sample several completions for the same prompt, score them, turn each score into a group-relative advantage, and update the policy while controlling drift from the reference behavior. The serving-style risk is that reward bugs become training bugs, so the reward contract belongs in version control before the GPU run starts.

def grpo_batch(prompt, policy, reward_fn, group_size=8):
    completions = [policy.generate(prompt) for _ in range(group_size)]
    rewards = torch.tensor([reward_fn(prompt, y) for y in completions])
    advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-6)
    return list(zip(completions, rewards.tolist(), advantages.tolist()))

1. Choose the reward first: prefer deterministic tests for math, code, schema, and format before adding a learned judge.
2. Inspect the group: for each prompt, log all sampled completions, rewards, and advantages so reward hacking is visible.
3. Track policy drift: monitor KL, response length, pass rate, and held-out quality slices; a higher reward can still mean worse user behavior.
4. Handle length carefully: modern trainer docs call out length-bias issues in GRPO-style losses, so record the loss variant and reward scaling in every run.

Sources: Direct Preference Optimization, Hugging Face TRL DPOTrainer, DeepSeekMath GRPO paper, Hugging Face TRL GRPOTrainer.

RAG quality depends on chunking, embedding choice, retrieval recall, reranking, context packing, citation discipline, and answer evaluation. The core failure is not just hallucination; it is often retrieving the wrong evidence or burying the right evidence.

Implementation Drill: Chunk For Retrieval, Not Storage

A chunk is a retrieval unit, citation unit, and prompt-packing unit at the same time. LangChain's recursive splitter starts from larger separators such as paragraphs and lines before falling back to smaller ones, while LlamaIndex's semantic splitter chooses sentence breakpoints from embedding similarity. Treat both as hypotheses to evaluate: fixed windows are easy to reproduce, recursive splitting preserves simple structure, and semantic splitting can keep topic shifts cleaner but adds embedding cost and more moving parts.

chunk_eval_case = {
    "question": "What timeout does the upload API use?",
    "gold_source_id": "manual.md#upload-timeouts",
    "splitter": "recursive | sentence | semantic",
    "chunk_size": 800,
    "overlap": 120,
    "metrics": ["recall_at_20", "citation_hit", "answer_supported"],
}

1. Keep metadata attached: every chunk should carry source id, section path, offsets, and version so retrieval results can become stable citations.
2. Evaluate boundaries: inspect misses where the answer spans two chunks, because more overlap may help more than a larger top-k.
3. Compare splitters: run the same labeled questions through fixed, recursive, and semantic splits before standardizing on one corpus policy.
4. Re-index deliberately: changing chunk size, overlap, parser, or embedding model creates a new retrieval dataset that needs its own eval report.

Sources: Retrieval-Augmented Generation, LangChain recursive text splitter, LlamaIndex semantic chunking.

Query transformation can bridge a user's short or vague wording to the terms used in the corpus. HyDE does this by asking an LLM to generate a hypothetical document or answer, embedding that generated text, and retrieving nearby real documents. The paper's important caveat is that the generated text can contain false details; the dense retrieval step should ground the search in the corpus, and the final answer should cite only retrieved source documents.

hyde_retrieval = {
    "user_query": "How long can upload jobs run?",
    "hypothetical_doc": llm_generate_search_probe(user_query),
    "retrieval_vector": embed(hypothetical_doc),
    "retrieved_sources": vector_index.search(retrieval_vector, top_k=20),
    "answer_context": rerank_and_pack(retrieved_sources),
    "citation_rule": "cite retrieved source text, never the hypothetical doc",
}

1. Use on semantic gaps: try HyDE when user questions are brief, conceptual, multilingual, or phrased differently from the source corpus.
2. Keep a baseline: compare raw-query retrieval, HyDE retrieval, and multi-query expansion on the same gold source ids before adopting it.
3. Do not cite the probe: the hypothetical document is an indexing aid, not authoritative context for the generator.
4. Budget the extra call: query transformation adds generation latency and another prompt surface, so it needs eval and observability like any other RAG stage.

Sources: HyDE paper, LangChain HyDE retriever docs, LlamaIndex HyDE query transform docs.

A retriever is only comparable across runs when the embedding contract is explicit. Record the embedding model, output dimension, normalization policy, distance metric, chunker version, metadata fields, and corpus snapshot before indexing. OpenAI's embedding docs frame embeddings as vectors for search and clustering, while the Retrieval API guide emphasizes semantic search over application data; the production lesson is that a changed embedding model or dimension is a new retrieval dataset, not a harmless implementation detail.

embedding_contract = {
    "embedding_model": "text-embedding-3-small",
    "dimensions": 1536,
    "normalized": True,
    "distance": "cosine",
    "chunker_version": "recursive-v2:800:120",
    "corpus_snapshot": "docs-2026-07-03",
    "metadata": ["source_id", "section_path", "offsets", "updated_at"],
}

1. Embed query and corpus consistently: do not compare vectors from different models, dimensions, tokenization policies, or normalization choices.
2. Store source metadata beside vectors: retrieval ids should lead back to stable source text, offsets, and document versions for citation and debugging.
3. Re-evaluate after reindexing: any change to model, chunking, dimensions, distance metric, or corpus parser should rerun recall and citation-hit checks.
4. Keep raw text recoverable: vectors are retrieval features, not evidence; the answer should cite the underlying chunk or document.

Sources: Retrieval-Augmented Generation, OpenAI embeddings guide, OpenAI Retrieval API guide.

Small chunks can lose the surrounding section, product, date, or entity that made them meaningful. Contextual retrieval fixes that by prepending a short document-aware note to each chunk before embedding and indexing. Anthropic describes pairing this with contextual BM25 and reranking; the important engineering rule is to evaluate it as a new index, not as harmless metadata. The added context changes both sparse terms and dense vectors, so it can improve recall while also introducing misleading boilerplate if the context generator is sloppy.

contextual_chunk = {
    "source_id": "handbook.md#pto-rollover",
    "raw_chunk": "Unused days roll over until March 31.",
    "context_note": "This chunk is from the PTO policy for US full-time employees.",
    "indexed_text": context_note + "\n\n" + raw_chunk,
    "visible_citation": raw_chunk,
}

1. Keep provenance clean: store generated context separately from source text so citations quote the original document, not the enrichment note.
2. Compare indexes: run raw chunks, contextual chunks, contextual BM25, and reranked candidates against the same gold evidence ids.
3. Inspect regressions: look for cases where a generic section note makes many chunks look relevant or where stale document summaries contaminate fresh chunks.
4. Cache deliberately: context generation can be expensive at ingest time, so version the prompt, model, source document hash, and generated note.

Sources: Anthropic contextual retrieval, Claude contextual embeddings cookbook, LangSmith RAG evaluation tutorial.

Dense retrieval is good at semantic matches, while lexical retrieval is hard to replace for exact names, ids, symbols, acronyms, and code. A hybrid first stage can run both searches, combine their ranked lists, then send the fused shortlist to a reranker. Reciprocal Rank Fusion is a practical default because it uses rank positions instead of assuming BM25 scores and embedding similarities are calibrated to the same scale.

def reciprocal_rank_fusion(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return [doc_id for doc_id, _ in sorted(
        scores.items(), key=lambda item: item[1], reverse=True
    )]

sparse = bm25.search(query, top_k=50)
dense = vector_index.search(embed(query), top_k=50)
candidates = reciprocal_rank_fusion([sparse, dense])[:50]
context = cross_encoder.rerank(query, candidates)[:6]

1. Evaluate by slice: compare dense-only, lexical-only, fused, and fused-plus-rerank results on acronym, identifier, paraphrase, and long-question cases.
2. Do not mix raw scores blindly: if the retrievers produce different score ranges, fuse by rank or use an explicitly normalized method.
3. Keep HyDE gated: hypothetical-document query expansion can improve zero-shot dense recall, but its generated text adds latency and may inject false details before corpus grounding.

Lab Prompt: Run A Retrieval Ablation Before Changing The Generator

Treat each retrieval stage as a candidate system with the same labeled questions and gold chunk ids. LlamaIndex's retrieval-evaluation docs call out ranking metrics such as hit rate, MRR, precision, recall, AP, and NDCG; Qdrant's hybrid query docs show fusion as a first-stage retrieval pattern; Sentence Transformers frames reranking as a second stage after fast retrieval. Put those together before blaming the LLM: measure whether dense search, sparse search, rank fusion, or reranking actually moves the gold evidence closer to the top.

rrf_ids = reciprocal_rank_fusion([sparse_ids, dense_ids])[:50]
retrieval_runs = {
    "dense_top_50": dense_ids,
    "sparse_top_50": sparse_ids,
    "rrf_top_50": rrf_ids,
    "reranked_top_10": rerank(query, rrf_ids[:50])[:10],
}

report = {
    name: retrieval_metrics(ids, gold_ids)
    for name, ids in retrieval_runs.items()
}

1. Hold the corpus fixed: use the same chunks, metadata, and gold ids for every candidate retriever.
2. Compare ranks, not anecdotes: report hit rate and MRR for each stage, then inspect only the cases that changed.
3. Gate rerank cost: promote a reranker only when its quality lift is worth the added candidate-pair latency.

Sources: Qdrant hybrid queries, Sentence Transformers retrieve and rerank, LlamaIndex retrieval evaluation, HyDE paper.

Dense retrieval turns questions and passages into vectors, then asks an index for nearest neighbors. DPR showed that a dual-encoder retriever can be a strong first stage for open-domain QA, while the original RAG recipe combined a neural retriever with a non-parametric passage index. In production, the index choice is an engineering contract: exact flat search is the quality baseline, and approximate indexes earn their keep only if they preserve enough recall at the latency and memory budget.

def recall_at_k(exact_ids, candidate_ids, k):
    gold = set(exact_ids[:k])
    found = set(candidate_ids[:k])
    return len(gold & found) / max(1, len(gold))

index_eval = {
    "baseline": "IndexFlatIP or IndexFlatL2",
    "candidate": "IVF, HNSW, PQ, or a managed vector index",
    "metrics": ["recall@10_vs_flat", "p95_search_ms", "bytes_per_vector"],
    "slices": ["exact_id", "acronym", "paraphrase", "long_question"],
}

1. Normalize deliberately: inner product and cosine search are only equivalent when embeddings are normalized.
2. Train approximate indexes: IVF and product-quantized indexes need representative data before their latency numbers mean much.
3. Measure against flat: tune index parameters by recall relative to exact search, not just by faster p95 latency.
4. Keep the reranker honest: a reranker cannot recover evidence that the first-stage index never returns.

Implementation Drill: Tune ANN Parameters Against A Flat Oracle

Approximate nearest-neighbor indexes are not a one-time infrastructure choice; their knobs are part of retrieval quality. Faiss documents flat indexes as the exact-result baseline, while index families such as IVF, HNSW, and PQ trade recall, memory, build time, and search latency. Keep a small labeled query set and sweep the index parameters against IndexFlatIP or IndexFlatL2 before blaming the embedding model or reranker.

ann_sweep = [
    {"index": "IVF4096,Flat", "params": {"nprobe": 8}},
    {"index": "IVF4096,Flat", "params": {"nprobe": 32}},
    {"index": "HNSW32,Flat", "params": {"efSearch": 64}},
    {"index": "HNSW32,Flat", "params": {"efSearch": 128}},
]

for candidate in ann_sweep:
    ids, latency_ms = search_with(candidate, queries, top_k=20)
    report(candidate, {
        "recall_vs_flat@20": recall_against_flat(ids, flat_ids, k=20),
        "p95_search_ms": percentile(latency_ms, 95),
        "index_bytes": measure_index_bytes(candidate),
    })

1. Sweep one family at a time: tune IVF's nprobe, HNSW's search breadth, or PQ compression while holding embeddings, chunks, and top-k fixed.
2. Use real slices: include exact ids, rare terms, paraphrases, and long questions so a fast average does not hide hard-query recall loss.
3. Report the Pareto point: choose the smallest latency and memory profile that still meets the recall target against the flat oracle.
4. Re-run on index changes: retraining IVF, changing normalization, adding PQ, or switching embedding models invalidates old recall numbers.

Sources: Dense Passage Retrieval, Retrieval-Augmented Generation, Faiss documentation, Faiss index guidelines.

Dense or hybrid retrieval should usually optimize for recall: pull a wide candidate set that probably contains the answer. A reranker then scores each query-document pair more carefully and sends only the best evidence into the prompt. Cross-encoders are slower than embedding lookup because they jointly encode the query and candidate text, but that extra interaction is useful when the right chunk appears in the top 50 and still loses to nearby distractors.

candidates = vector_index.search(query, top_k=50)
scored = [
    (cross_encoder.score(query, chunk.text), chunk)
    for chunk in candidates
]
context = [chunk for score, chunk in sorted(scored, reverse=True)[:6]]

1. Measure both stages: track recall@50 before reranking and nDCG or hit rate after reranking.
2. Budget latency: rerank only the candidate count your p95 latency can afford; batch candidate pairs when possible.
3. Keep citations stable: reranking should reorder chunk ids, not rewrite or detach evidence from its source metadata.

Sources: Sentence Transformers retrieve and rerank, Sentence Transformers reranker docs.

Bigger context windows do not remove the need for retrieval design. The Lost in the Middle results show that models can be worse at using evidence placed deep inside a long prompt than evidence near the beginning or end. Treat context packing as a tested policy: deduplicate chunks, preserve citations, keep the highest-value evidence visible, and choose whether to stuff, compact, or refine across chunks based on the answer type.

context_pack = [
    system_instruction,
    query,
    top_evidence_with_citations,
    supporting_evidence,
    conflict_or_uncertainty_notes,
    answer_format,
]

1. Position test: run the same golden question with the key chunk first, middle, and last; fail the build if groundedness changes materially.
2. Budget test: compare top-k stuffing with a compact/refine synthesis path when retrieved evidence exceeds the prompt budget.
3. Citation test: require every factual sentence that depends on retrieval to point back to a chunk id or source URL.

Lab Prompt: Run A Position-Sensitivity Ablation

The Lost in the Middle paper tested whether models use relevant information equally well across long contexts and found that position can change performance. Turn that into a RAG regression test by holding the retrieved chunks fixed, moving the gold chunk to different prompt positions, and grading the same answer contract each time. If the answer only succeeds when evidence is first or last, the packer needs a salience policy instead of a larger top-k.

def pack_with_gold_position(chunks, gold_id, position):
    gold = next(chunk for chunk in chunks if chunk.id == gold_id)
    distractors = [chunk for chunk in chunks if chunk.id != gold_id]
    if position == "front":
        ordered = [gold] + distractors
    elif position == "middle":
        midpoint = len(distractors) // 2
        ordered = distractors[:midpoint] + [gold] + distractors[midpoint:]
    elif position == "end":
        ordered = distractors + [gold]
    else:
        raise ValueError("position must be front, middle, or end")
    return render_context(ordered)

for position in ["front", "middle", "end"]:
    prompt = build_rag_prompt(query, pack_with_gold_position(chunks, gold_id, position))
    grade(position, model.generate(prompt), expected_citation=gold_id)

1. Hold retrieval constant: use the same chunks and source ids so the test isolates packing position, not recall.
2. Grade citations: require the answer to cite the gold chunk, because a plausible answer without evidence can hide position failure.
3. Promote salient evidence: put decisive, recent, or high-confidence chunks in stable prompt slots instead of relying on arbitrary rank order.

Sources: Lost in the Middle, LlamaIndex response synthesizers.

A RAG answer can fail because the retriever missed the right evidence, ranked noisy chunks above useful chunks, or because the generator ignored the evidence it received. Keep the eval record explicit so each stage has its own signal before you tune embeddings, chunking, prompts, or models.

rag_eval_case = {
    "query": str,
    "expected_evidence": list[str],
    "retrieved_chunks": list[str],
    "answer": str,
    "citations": list[str],
    "grades": {
        "context_recall": float,
        "context_precision": float,
        "faithfulness": float,
        "answer_relevance": float,
    },
}

1. Context recall: did retrieval include the evidence needed to answer?
2. Context precision: did high-ranked chunks stay focused, or did they waste context budget?
3. Faithfulness: does the answer stay supported by retrieved text instead of importing unsupported claims?
4. Answer relevance: does the answer actually resolve the user query after grounding checks pass?

Implementation Drill: Compute Hit Rate And MRR Before Judging Answers

Before asking an LLM judge whether the final answer is faithful, check whether retrieval put the gold evidence in front of the generator. LlamaIndex's retrieval-evaluation docs use ranking metrics such as hit rate and mean reciprocal rank, while RAGAS separates context recall and context precision from answer-level faithfulness. That separation keeps a weak retriever from being hidden by a lucky generator.

def retrieval_metrics(retrieved_ids, expected_ids):
    expected = set(expected_ids)
    first_hit_rank = None
    for rank, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in expected:
            first_hit_rank = rank
            break

    return {
        "hit_rate": 1.0 if first_hit_rank is not None else 0.0,
        "mrr": 0.0 if first_hit_rank is None else 1.0 / first_hit_rank,
    }

1. Evaluate retrieval alone: run hit rate, MRR, recall, or nDCG against expected chunk ids before adding the generator.
2. Keep ids stable: store document id, chunk id, source URL, and tokenizer/chunker version so metric changes can be traced to a pipeline change.
3. Slice hard cases: report exact-id, acronym, paraphrase, multi-hop, and stale-document queries separately instead of averaging them away.

Evaluation Drill: Calibrate The RAG Judge

Automated RAG judges are useful only when their contract is explicit. Ragas decomposes the task into retriever-facing context precision and context recall plus generator-facing faithfulness, while ARES evaluates context relevance, answer faithfulness, and answer relevance with lightweight judges calibrated by a small human-labeled set. For a course project or production launch, keep a tiny audited slice beside the automated scores so judge drift does not become a hidden regression.

judge_audit = {
    "case_id": "rag-042",
    "query_type": "multi-hop",
    "expected_context_ids": ["policy:7", "faq:3"],
    "retrieved_context_ids": ["faq:3", "blog:9", "policy:7"],
    "answer_claims": [
        {"text": "Refunds require approval after 30 days.", "supported": True},
        {"text": "Enterprise plans always bypass approval.", "supported": False},
    ],
    "human_grade": {
        "context_recall": 1.0,
        "context_precision": 2 / 3,
        "faithfulness": 0.5,
    },
}

1. Pin the rubric: define relevance, support, and answer usefulness before collecting judge scores.
2. Audit by slice: inspect single-hop, multi-hop, stale-document, and adversarial queries separately because average faithfulness can hide one broken category.
3. Compare to humans: sample automated passes and failures each release, then track judge agreement before trusting a score movement.
4. Keep evidence ids: store stable chunk ids with the answer so a later reviewer can reproduce whether each claim was actually grounded.

Sources: RAGAS, Ragas context precision, Ragas context recall, Ragas faithfulness, ARES, LlamaIndex retrieval evaluation, OpenAI Evals.

Agents add value when a task requires multi-step tool use, checking external state, retrying after failures, or maintaining a work log. They add risk when a simple one-shot call would be easier to test and control.

Implementation Drill: Define The Control Plane Before Adding Tools

A production agent needs more than a list of functions. Decide which actions require approval, which steps can be retried automatically, how state is persisted between turns, and where execution can pause for a human. OpenAI's Agents SDK separates orchestration, tool execution, approvals, guardrails, and tracing; MCP standardizes connections to external systems; LangGraph interrupts show the same reliability pattern from the workflow side: pause, persist state, then resume from an explicit decision.

agent_control_plane = {
    "state_store": "durable run state, not only chat history",
    "approval_required": ["send_email", "delete_file", "purchase", "public_post"],
    "auto_retry": {"search_docs": 2, "read_file": 1},
    "interrupt_points": ["before_external_action", "after_ambiguous_tool_result"],
    "trace_fields": ["model_call", "tool_call", "guardrail", "handoff", "human_decision"],
}

1. Classify tools: read-only, reversible write, irreversible write, and external communication tools should not share the same execution policy.
2. Persist before waiting: if the agent asks for approval, save the run state and pending action so resume does not depend on hidden context.
3. Trace decisions: log the model proposal, validated arguments, guardrail result, human decision, and final tool result for audits and regression tests.
4. Test recovery: simulate tool timeouts, denied approvals, stale state, and duplicate resumes before trusting the loop in production.

Implementation Drill: Treat Tool Calls As Validated Transactions

A tool schema helps the model propose structured arguments, but the application still owns validation, authorization, execution, and result handling. Before a tool call touches external state, parse the arguments against the declared schema, apply a permission policy, record an idempotency key, and store the result as a typed observation rather than untrusted free text. MCP uses tool metadata and input schemas to expose executable capabilities, while OpenAI function calling uses JSON-schema-style tool definitions to let the model request application actions.

tool_transaction = {
    "tool_name": "create_calendar_event",
    "arguments": validated_json,
    "permission": "approved | denied | read_only",
    "idempotency_key": request_id + ":" + step_id,
    "result": typed_tool_result,
    "observation_visible_to_model": summarized_result,
}

1. Validate twice: check model-produced arguments before execution and tool-produced results before feeding them back into the agent state.
2. Separate permission from parsing: a schema-valid request can still require approval, tenant checks, rate limits, or a dry-run preview.
3. Make writes idempotent: retries after a timeout should not create duplicate emails, tickets, purchases, or database mutations.
4. Constrain observations: strip secrets, huge payloads, and prompt-injection text from tool results before the model sees them.

Sources: OpenAI Agents SDK guide, OpenAI function calling guide, OpenAI Agents SDK tracing, OpenAI Agents SDK guardrails, Model Context Protocol introduction, MCP tools specification, LangGraph interrupts.

Tool descriptions are part of the agent's control surface. MCP tool definitions expose names, descriptions, input schemas, optional output schemas, annotations, and execution metadata, while OpenAI Structured Outputs distinguishes valid JSON from schema-adherent JSON. Treat those contracts as test fixtures: the model may propose a call, but the client should still validate arguments, policy, outputs, and trace identity before continuing the run.

tool_contract = {
    "name": "tickets.create",
    "input_schema": {"type": "object", "required": ["title", "priority"]},
    "output_schema": {"type": "object", "required": ["ticket_id", "status"]},
    "side_effect": "write",
    "approval_policy": "human_required",
    "trusted_metadata_source": "internal_tool_registry",
}

1. Schema test: reject missing required fields, extra fields when disallowed, unsupported enum values, and malformed empty-argument tools.
2. Metadata test: verify tool names are stable, unique, and routeable; do not let untrusted descriptions or annotations become authorization policy.
3. Policy test: map read-only, reversible write, external communication, and irreversible write tools to separate approval and retry rules.
4. Output test: validate structured tool results against the output schema before summarizing them back into model-visible state.
5. Resume test: if a call pauses for review, persist the pending call id, validated arguments, policy decision, and exact resume point.

Sources: MCP tools specification, OpenAI Structured Outputs guide, LangGraph interrupts.

Tool use creates at least three boundaries: the model reads tool metadata, the client executes a proposed call, and a downstream system accepts credentials or writes state. The MCP tool specification treats tool annotations as untrusted unless they come from trusted servers, recommends human confirmation for sensitive invocations, and assigns servers and clients separate duties for input validation, access control, output sanitization, timeouts, and audit logs. OpenAI's Agents SDK guardrails give the same idea a runtime shape: validate or block tool input before execution and validate tool output before the agent consumes it.

tool_trust_boundary = {
    "metadata": "display and route, but do not treat as policy",
    "arguments": "schema + permission + tenant checks before execution",
    "credentials": "audience-bound token, never generic passthrough",
    "result": "typed, size-capped, sanitized observation",
    "audit": "user, client_id, tool_name, scopes, call_id, decision",
}

1. Review metadata like input: tool names, descriptions, annotations, icons, and resource links can shape model behavior, so only trusted registries should influence high-risk tool choice.
2. Bind authorization to the resource: MCP authorization requires clients to request tokens for the intended resource and servers to reject tokens not issued for them.
3. Forbid token passthrough: do not let an MCP server relay a client token to a downstream API; use a separate downstream token with explicit audience and scope.
4. Gate side effects before execution: use blocking guardrails, approval prompts, or dry-run previews for writes, spending, communication, and public actions.
5. Sanitize results after execution: tool output is data, not developer instruction; cap payloads, redact secrets, validate structured fields, and preserve the raw result outside the prompt when possible.

Sources: MCP tools specification, MCP authorization specification, MCP security best practices, OpenAI Agents SDK guardrails, NSA MCP security design considerations.

Tool-using agents are easier to evaluate when every step has a typed record: user goal, model decision, tool name, validated arguments, tool result, model observation, retry reason, and final answer. ReAct-style loops make the reasoning/action alternation explicit, while Toolformer highlights that useful tool use includes deciding when to call a tool, what arguments to pass, and how to fold the result back into generation.

agent_trace_event = {
    "request_id": str,
    "step": int,
    "call_id": optional(str),
    "decision": "answer" | "call_tool" | "ask_clarifying_question",
    "tool_name": optional(str),
    "arguments_valid": bool,
    "tool_is_error": optional(bool),
    "observation_used": bool,
    "observation_sanitized": bool,
    "retry_reason": optional(str),
}

1. Unit-test schemas: reject missing required fields, unknown tools, and unsafe argument shapes before execution.
2. Preserve call identity: keep the model's tool-call id or MCP request id with the result so parallel calls, retries, and late failures cannot be attached to the wrong step.
3. Classify errors: separate invalid arguments, unknown tools, server failures, and business-logic failures; only retry cases that are actually retryable.
4. Eval trajectories: grade not only the final answer, but whether the agent called the right tool at the right time.
5. Sanitize observations: treat tool output as untrusted input before the model sees it; redact secrets, cap payload size, and keep external instructions as quoted evidence rather than system policy.
6. Log recovery: capture retries and fallbacks so failures become debuggable traces instead of opaque conversations.

Implementation Drill: Replay Tool Results By Id

Tool use becomes hard to debug when results are stored only as text in the chat transcript. Store each proposed call, validated arguments, execution result, and model-visible observation under the same call id. OpenAI function calling returns a call_id that is echoed when sending the tool result back to the model; MCP tool calls similarly carry request identity, named tools, JSON arguments, result content, and an isError flag. That identity is what makes trace replay, duplicate suppression, and parallel tool calls testable.

tool_result_record = {
    "call_id": "call_123",
    "tool_name": "search_docs",
    "validated_arguments": {"query": "KV cache"},
    "raw_result": stored_outside_prompt,
    "is_error": False,
    "model_observation": "Top source ids: doc_7, doc_2",
}

1. Replay check: a saved trace should reconstruct the same model-visible observations without re-running external tools.
2. Duplicate check: if the same write call is retried, the call id or idempotency key should prevent a second external mutation.
3. Injection check: tool text can contain hostile instructions; the trace should mark it as data from a tool, not trusted developer policy.

Sources: ReAct, Toolformer, OpenAI function calling guide, MCP tools specification.