Skip to main content

Chapter 6 — Evaluation

Exam domain: Evaluation — 7%

Purpose​

Learn to prove whether an LLM system is useful, safe, and production-ready through task-aligned metrics, benchmarks, and error analysis.

Learning objectives​

  • Choose metrics that match the product decision.
  • Explain perplexity, BLEU, ROUGE, exact match, retrieval metrics, and LLM-as-a-judge.
  • Design human review and error-analysis workflows.
  • Benchmark systems fairly across hardware and deployments.
  • Recognize metric gaming and evaluator bias.

1. Evaluation starts with the decision​

Ask what decision the measurement will support: select a model, approve a deployment, detect a regression, or compare optimization variants. Then define users, tasks, harm tolerance, quality bar, latency/cost limits, and representative data slices.

2. Core metric families​

  • Perplexity: measures how surprised a language model is by held-out text; useful for language modeling but not a direct measure of chat helpfulness.
  • BLEU/ROUGE: overlap metrics; useful in some constrained generation settings but insufficient for open-ended quality or factuality.
  • Exact match/F1: suitable for structured extraction or short-answer tasks with clear ground truth.
  • Retrieval recall, precision, MRR, nDCG: assess whether RAG retrieval finds and ranks the right evidence.
  • System metrics: latency, throughput, error rate, token cost, and GPU memory.

3. Human evaluation​

Human reviewers can assess correctness, usefulness, tone, policy compliance, and groundedness. Use clear rubrics, blinded comparisons where possible, representative samples, and disagreement analysis. Reviewers should see source evidence when judging a RAG answer.

4. LLM-as-a-judge​

A judge model can scale evaluation but is itself a model with biases and blind spots. Validate it against human judgments, use clear rubrics, hide irrelevant identity/model labels, check position bias, and keep adversarial cases. A judge score is evidence, not ground truth.

5. Error analysis​

Group failures by pattern: retrieval miss, unsupported claim, instruction following, formatting, safety, long-context, language, tool call, or latency failure. A useful report includes examples, frequency, severity, root-cause hypothesis, owner, and next experiment.

Mini-lab​

Create a 30-item evaluation set for a hypothetical RAG assistant. Include answerable, unanswerable, ambiguous, adversarial, and outdated-information questions. For each, specify expected behavior, evidence, quality score, and latency target.

Study checkpoint​

  • I can state when perplexity is useful and when it is misleading.
  • I can distinguish retrieval evaluation from answer evaluation.
  • I can describe a defensible LLM-as-a-judge setup.
  • I can turn a vague “quality issue” into an error-analysis category.

Further learning resources​

Reading and documentation​

  1. Ragas documentation
  2. EleutherAI lm-evaluation-harness
  3. Hugging Face Evaluate documentation
  4. NVIDIA NeMo Evaluator
  5. HELM benchmark

Videos​

Suggested study rhythm​

Spend one focused session learning the concepts, a second session doing the mini-lab, and a third session explaining the checkpoint items without notes. Do not advance simply because the vocabulary is familiar; advance when you can select a method, state its tradeoff, and diagnose a realistic failure case.