Module E10
Evaluation, experiments, and observability
Produce reproducible evidence about a predictive or generative system and diagnose its failures.
Why this module
Produce reproducible evidence about a predictive or generative system and diagnose its failures.
What you produce
A versioned evaluation harness. Assess the output against the unit outcomes, explanation, safe handling, and independent transfer. Do not require public sharing or use of real sensitive records.
Chapters and units
E10-C01: Metrics and experiment design
Define success and a simpler comparator for the chosen system; Keep tuning separate from final evaluation
- E10-U01 Task metrics, labels, and baselines Define success and a simpler comparator for the chosen system
- E10-U02 Splits, uncertainty, and representative slices Keep tuning separate from final evaluation
E10-C02: Judgment and diagnostic evidence
Calibrate judgments using human-reviewed examples; Trace a failure to data, prompt, model, tool, or code
- E10-U03 Human rubrics and automated judges Calibrate judgments using human-reviewed examples
- E10-U04 Traces, versions, and regression checks Trace a failure to data, prompt, model, tool, or code
E10-C03: Comparison and release decisions
Report raw counts, tradeoffs, and sample limitations; Make a supported go/no-go recommendation
- E10-U05 Comparative experiment and limitations Report raw counts, tradeoffs, and sample limitations
- E10-U06 Evaluation report and release recommendation Make a supported go/no-go recommendation