evals
Evidence Is What Someone Can Verify Later
An eval result becomes evidence only when the party that needs to verify a claim can retrieve it, interpret it, and verify it. Production alone is not enough. Retention alone is not enough.
The Eval Pack Belongs to the Class
An eval pack is the reusable unit of evaluation for an agent class. It contains scenarios, scorers, thresholds, expected evidence, and obligation mappings — the content that turns a test suite into traceable evidence.
Drift Is When the Queue Does Not Know
Change routing works only for visible change. Drift is the movement that invalidates evidence without producing a change record.
Baseline Inheritance Is How Agent Evaluation Scales
An instance inherits its class baseline only while it stays inside the boundary the baseline was proven against. The hard part of fleet evaluation is detecting the moment that boundary has been crossed.
Evaluation Should Follow Change
Calendar-driven eval cadence wastes capacity on stable systems and misses risk on changing ones. Change-routed evaluation matches eval work to what actually changed.
The Next Eval Is the One with the Most Evidence at Risk
The next eval should be the one where delay puts the most load-bearing evidence at risk.
The Agent Is Not The Unit. The Agent Class Is.
Per-agent evaluation fails at fleet scale because the unit of review is wrong. The reviewable unit is the agent class: a shared pattern of purpose, tools, data access, autonomy, and risk surface.
Why Per-Agent Evaluation Breaks at Fleet Scale
Most evaluation systems assume a single agent. At fleet scale, the question shifts from whether one agent passed to where limited evaluation capacity should be spent now.
Choosing Your Eval Architecture
The question is not which eval tool. The question is what kind of eval infrastructure your system actually needs. Three architectures, three failure modes, and how they compose into an evidence pipeline.
Drift Detection Patterns for Production Agents
Your agent is still answering. That does not mean it is still behaving the same way. Five drift classes, three detection layers, and the patterns that catch regression before your customers do.
Building an Eval Harness That Survives Production
Most eval harnesses die the same way. Five structural decisions separate the ones that survive production from the ones that quietly rot.
What Should an AI System Actually Prove?
You diagnosed the problem five different ways. Now build the answer. The proof loop: obligation, control, evaluation, evidence, response.
Drift Is the Default
Your agent worked yesterday. That is not a promise about today. Model updates, prompt changes, and shifting inputs cause silent behavioral regression that traditional monitoring doesn't catch.
The Eval Gap: Why Your Agent Works in Staging and Breaks in Production
Your benchmarks are passing. Your agent is failing. Most evals measure isolated performance under controlled conditions while production failure comes from distribution shift, tool-chain errors, and changing reality.