Skip to main content

Explore Problems

Certifications Pricing Discuss

LLM Evaluation & Observability

Course Contents

The Eval-Driven Development Mindset

Why Evaluating LLMs Is Uniquely Hard

Offline vs Online Evaluation

Building Your First Eval Set

Eval Anti-Patterns (Vibes-Based Testing)

Functional Correctness & Exact Match

Similarity Metrics (Embedding, BLEU/ROUGE — and Their Limits)

Perplexity & Entropy (Concepts)

Task-Specific Metrics (Classification, Extraction, Summarization)

Reference-Based vs Reference-Free

Statistical Rigor: Sample Size & Confidence

The Idea: Models Grading Models

Pointwise vs Pairwise / Comparative Judging

Writing Judge Prompts & Rubrics

Judge Biases & How to Calibrate

Human-in-the-Loop Spot Checks

Look at Your Data (The #1 Rule)

Open Coding & Failure Taxonomies

From Failures to Targeted Evals

The Analyze → Measure → Improve Loop

Catching Regressions Before They Ship

Case Study: Debugging a Failing AI Feature

Tracing LLM & Agent Calls

Evaluating Agents: Trajectories, Tool Correctness & Step Efficiency

Observability Tools: LangSmith, Langfuse, Phoenix, Braintrust, Helicone

Online Evals & Guardrail Metrics

Detecting Drift & Degradation

Capturing & Using User Feedback

Closing the Loop: Production → Dataset

From Failures to Targeted Evals

From Failures to Targeted Evals on CodePeet.

Loading...

Loading content...