Skip to main content

Chapter 9 — Production Monitoring and Reliability

Exam domain: Production Monitoring and Reliability — 7%

Purpose​

Learn to operate an LLM system after launch: observe its behavior, detect regressions, respond to incidents, and improve safely.

Learning objectives​

  • Define service, model, retrieval, quality, safety, and cost signals.
  • Use logs, metrics, traces, and version metadata together.
  • Design a regression and release-monitoring loop.
  • Differentiate online monitoring from offline evaluation.
  • Plan incident response and rollback.

1. Measure the whole system​

LLM reliability includes availability and latency, but also answer quality, grounding, safety, retrieval success, tool reliability, token cost, and configuration drift. A server can be “up” while users receive misleading answers.

A practical dashboard usually includes request rate, p50/p95/p99 latency, error/timeout rate, tokens in/out, queue time, GPU utilization, cache pressure, retrieval latency, empty-retrieval rate, refusal rate, safety events, feedback, and cost. Segment by model version, prompt version, tenant, language, task, and traffic route where appropriate.

2. Logs, metrics, and traces​

  • Logs explain discrete events and errors.
  • Metrics show aggregates and trends suitable for SLOs/alerts.
  • Traces follow a request across retrieval, guardrails, tools, model calls, and downstream services.

Do not log raw sensitive prompts or secrets by default. Use privacy-aware event schemas, access controls, retention rules, and redaction.

3. Version everything that changes behavior​

Record model/adapter, tokenizer, prompt template, retrieval index/corpus version, embedding model, reranker, guardrail policy, tool schema, serving image, and runtime configuration. Without version metadata, an incident cannot be reproduced reliably.

4. Continuous benchmarking​

Run fixed offline regression suites before releases and periodically after releases. Compare to a named baseline. Add real failure patterns to the suite after privacy/security review. Online feedback is valuable, but it is biased by who uses the product and what they choose to report.

5. Reliability operations​

Define SLOs and error budgets, use health/readiness checks, capacity planning, rate limits, backpressure, dependency timeouts, retries appropriate to idempotency, fallback behavior, and a tested rollback path. For an LLM service, a safe fallback might be a grounded search result, a narrower workflow, or a transparent “unable to answer” response—not necessarily another unconstrained model call.

Mini-lab​

Draft an incident runbook for “grounded-answer quality suddenly dropped.” Include dashboards to inspect, configuration versions to compare, a safe mitigation, user communication, rollback decision, and the regression test to add after resolution.

Study checkpoint​

  • I can distinguish logs, metrics, and traces.
  • I can name both classical service and LLM-specific reliability signals.
  • I can explain why versioning the model alone is insufficient.
  • I can design an alert that is actionable rather than noisy.

Further learning resources​

Reading and documentation​

  1. OpenTelemetry documentation
  2. Prometheus documentation
  3. NVIDIA NeMo Guardrails observability
  4. Google SRE Book
  5. NVIDIA Triton metrics documentation

Videos​

Suggested study rhythm​

Spend one focused session learning the concepts, a second session doing the mini-lab, and a third session explaining the checkpoint items without notes. Do not advance simply because the vocabulary is familiar; advance when you can select a method, state its tradeoff, and diagnose a realistic failure case.