Evaluations and Observability for Claude Apps
Build eval harnesses that catch regressions before they ship. Cover test-bank design, LLM-as-judge patterns, tracing and structured logging, cost and latency monitoring, and wiring prompt changes into CI so quality stays measurable as your Claude app grows.
View badge details
About This Course
Build eval harnesses that catch regressions before they ship: test-bank design, LLM-as-judge patterns, tracing + structured logging, cost + latency monitoring, and CI integration for prompt changes.
Course Curriculum
10 Lessons
Why evals for Claude apps
By the end of this lesson you will know why systematic evals separate a demo Claude app from a shipped one — they catch regressions when you change a prompt, upgrade a model version, or refresh a corpus — and the three-layer eval stack every production app grows into: a labeled test bank, a scoring layer (deterministic checks + LLM-as-judge), and CI integration that blocks a prompt PR on regression. Sets up L2 hands-on where you build a working scorecard.
Build an eval scorecard - Lab Exercises
By the end of this hands-on lab you will have built a four-case Claude eval scorecard covering the happy path, a boundary case, an adversarial-prompt case, and an out-of-scope case — with deterministic must_contain and must_not_contain checks per case that report a machine-parseable pass rate. Uses the PythonAI container + Claude API proxy. Introduces the eval-bank shape you'll extend in every remaining CLD-AI-109 lab with LLM-judge scoring and CI integration.
LLM-as-judge patterns
By the end of this lesson you will know how to design LLM-as-judge prompts that produce stable scores: structured verdicts via tool_use (yes/no + short rationale), per-criterion rubrics (relevance, tone, correctness — one call per criterion, not one mega-rubric), cross-validation against a small human-labeled sample to calibrate the judge's threshold, and defenses against self-affirmation bias when judge and generator use the same model tier. Sets up L4 hands-on where you wire a real judge.
LLM-judge harness - Lab Exercises
By the end of this hands-on lab you will have built an LLM-judge harness that scores one Claude answer against three semantic criteria (relevance, tone, correctness) via a separate judge Claude call using tool_use for structured verdicts, printed a per-criterion pass/fail scorecard with rationales, and observed how per-criterion prompts beat a single mega-rubric on stability. Uses the PythonAI container + Claude API proxy — measures cost overhead of the judge layer.
Regression testing on prompt changes
By the end of this lesson you will know how to wire regression tests around prompt and model changes — run the eval bank against version A (baseline) and version B (proposed change), compute a per-case delta, and block the merge on any case that passed in A and fails in B — plus how to slot the regression check into CI (GitHub Actions) so a prompt PR can't ship without a green scorecard. Sets up L6 hands-on where you compare two prompt versions.
Regression harness comparing 2 prompt versions - Lab Exercises
By the end of this hands-on lab you will have built a regression harness that runs a three-case eval bank against two system prompt versions in parallel, produces a per-case A-vs-B pass/fail matrix, and flags regressions (case passed in A but fails in B) as blocking. Includes an exit-code contract suitable for CI gating. Uses the PythonAI container + Claude API proxy. Prepares you for the L10 capstone that combines the full eval + tracing stack.
Tracing and structured logging
By the end of this lesson you will know how to instrument a production Claude app: ship a trace_id per user request that propagates across every internal call (Claude, tool servers, MCP servers), emit structured JSON logs with a stable schema (trace_id, tokens_in, tokens_out, latency_ms, cost_usd, stop_reason), and integrate with OpenTelemetry or Datadog for aggregation and alerting. Anchors the observability discipline used in L8 hands-on and reappears in CLD-AI-112.
Trace logger for Claude calls - Lab Exercises
By the end of this hands-on lab you will have wrapped an Anthropic SDK client with a structured-JSON trace emitter — every Claude call produces one log line carrying trace_id, tokens_in, tokens_out, latency_ms, computed cost_usd, cache-read/write counters, and stop_reason — ready for shipping into OpenTelemetry or Datadog. Uses the PythonAI container + Claude API proxy. Prepares the trace shape you'll aggregate in the L10 capstone scorecard.
Cost + latency dashboards
By the end of this lesson you will know which cost + latency metrics matter for a Claude app dashboard: P50 / P95 / P99 latency per agent role, cost per user request over time (with trendline alerts), tool-call success rates (429 storms surface here first), and token efficiency (output tokens per user request as a proxy for prompt bloat). Anchors the dashboard shape you'll wire into the L10 capstone scorecard and reappears in CLD-AI-112 incident response.
Full eval + monitoring stack capstone - Lab Exercises
By the end of this hands-on capstone you will have shipped the full CLD-AI-109 eval + monitoring stack — the four-case bank from L2, the LLM-judge from L4, the regression harness from L6, and structured traces from L8 — running end-to-end and producing an aggregate JSON scorecard (pass rate, P50 / P95 latency, cost per request) defensible in a CI gate. Uses the PythonAI container + Claude API proxy — this is the shape you'll paste into a real production repo.
This course includes:
- 24/7 AI Instructor Support
- Live Lab Environments
- 5 Hands-on Lessons
- Completion Badge
Earn Your Badge
Complete all lessons to unlock the Evaluations and Observability for Claude Apps achievement badge.
Evaluations and Observability for Claude Apps
Awarded on completion of CLD-AI-109. The holder can design eval banks, implement LLM-as-judge patterns, add structured tracing + cost/latency metrics, and gate deploys on eval regressions.
Skills You'll Earn
Complete all lessons in this course to earn this badge