Accuracy is the wrong metric
An agent that is right most of the time and unpredictable the rest of the time is not a good agent. It is an unmanaged one.
Every evaluation deck opens with an accuracy number. It is the wrong number. Accuracy tells you how often the system produced the right answer on a set of cases someone chose in advance. It tells you nothing about what happens on the cases nobody chose, and nothing at all about the cost of being wrong.
A support agent that resolves nine tickets out of ten looks excellent until you learn that the tenth is a refund it issued to the wrong account. A document extractor that is right ninety-five percent of the time is unusable if it cannot tell you which five percent to check. The distribution of errors matters more than the rate of them, and accuracy collapses that distribution into a single number that hides everything you needed to know.
What to measure instead
Replace the single score with four measures. Each one maps to a decision you will actually have to make about the system, which is the test of whether a metric is worth instrumenting.
| Measure | The question it answers | How you instrument it |
|---|---|---|
| 01Calibration | When the agent says it is confident, is it? Can it tell you which outputs to check? | Log a confidence signal per output, then sample and score against ground truth by confidence band. |
| 02Containment | What is the worst thing a single wrong answer can do before a human sees it? | Enumerate the write actions in scope and cap the blast radius of each one. |
| 03Escalation rate | How often does the agent correctly decline and hand off rather than guess? | Track handoffs as a first-class outcome, not a failure, and audit the ones it should have made. |
| 04Recovery time | When it does go wrong, how long until someone notices and reverses it? | Measure from the bad action to the corrective action, not from the complaint. |
A system that is right eighty percent of the time and knows which eighty is more valuable than one that is right ninety-five percent of the time and cannot tell you which.
Build the evaluation set from the mess
Most evaluation sets are built from clean examples because clean examples are easy to collect. That is exactly backwards. The cases that decide whether a system ships are the ambiguous ones, the half-complete records, the customer who asks three things in one message, the invoice in a format nobody has seen since 2019. Pull your evaluation set from real historical traffic, deliberately over-sampling the cases that a human escalated or got wrong.
- Include every case from the last year that triggered a complaint or a manual correction.
- Include the long tail: the formats, languages, and edge conditions that appear once a quarter.
- Include adversarial input, because if the agent is customer-facing, it will receive some.
- Keep a held-out slice that nobody prompts against, so you can tell tuning from overfitting.
Evaluation is a standing function, not a gate
The common failure is treating evaluation as something you do once before launch. Models change under you, upstream data changes, and the people using the system learn to phrase things differently. An evaluation suite that is not run on a schedule is a snapshot of a system that no longer exists.
Run the suite on every prompt change, every model version bump, and on a fixed cadence regardless. Then make one person accountable for the result. Evaluation with no owner produces dashboards nobody reads.
Want this graded for your own stack?
A systems audit runs your operation against exactly these dimensions and hands you the report.
Request a systems audit