Evaluation and observability

Evaluation and observability are related but answer different questions.

Evaluation before promotion

An evaluation case is an input with the expected outcome or criteria for success. Good case sets are built from real work: normal requests, edge cases, incomplete context, policy boundaries, and situations where the correct response is to refuse or wait for approval.

Evaluate the final result and the process that produced it. A response can read well while relying on stale data, selecting the wrong tool, exceeding its cost budget, or taking an unauthorized action.

Observability in production

Production traces and metrics show how the released system behaves under real conditions. They reveal changed source freshness, provider failures, latency, repeated policy denials, unusual spend, and failure patterns that a pre-release case set did not contain.

The improvement loop

When production reveals a meaningful failure, turn it into a new evaluation case. When an evaluation exposes a regression, inspect the trace to identify the cause. This creates a feedback loop between development evidence and operational evidence.

Next: Security and approvals.