How to evaluate an agent beyond its best demo

What the evaluation guide establishes

Anthropic’s January guide distinguishes an execution transcript from the final state left in the environment. It treats an agent result as the work of a model plus its harness. It also separates pass@k, success at least once across k attempts, from pass^k, success on all k trials. The two metrics reward different behavior.

Source: Anthropic: Demystifying evals for AI agents

Editorial interpretation

Synthetic example: prepare an invoice draft

This worked example is invented. No agent has been run and no customer invoice has been created. The fictional task is to save an invoice draft for client C-104, using eight hours at US$75 per hour from a supplied work log. The expected subtotal is US$600. Tax is explicitly excluded from this exercise.

The permitted actions are reading the synthetic client record and work log, then writing one draft file. Sending an invoice, changing client records and taking payment are outside the task. Success requires the correct client identifier, quantity, rate and subtotal in the saved artifact, with no duplicate draft or external send.

Prepare a known working reference artifact before testing. Have the checker read the saved fields instead of trusting the final chat message. If a tool reports a successful write but the expected file is absent, the task has not met its outcome criteria.

Editorial implications, not an independently measured result.

Editorial interpretation

Make the failures informative

Add a second case in which the work log has no client identifier. The expected behavior is to request clarification or leave the draft uncreated. A confident guess should fail this case even when the arithmetic is correct.

Add a third case that interrupts the save operation. On retry, inspect whether the system leaves one usable draft, two drafts or an uncertain state. This test is designed to expose duplicate creation and recovery behavior; it is not a benchmark result.

For each attempt, begin from a documented clean state and preserve the input, output artifact and intervention notes. Keep the grading rule stable while comparing candidates. If the rule is wrong, revise it openly and rerun the relevant cases instead of silently rescoring a favored example.

Editorial implications, not an independently measured result.

Editorial interpretation

Choose the metric for the decision

For this proposed workflow, report observed successful attempts alongside total attempts, then show correction minutes and duplicate drafts separately. A single combined score could conceal the exact failure that prevents deployment.

Retries may be acceptable when a reviewer selects one draft before sending. They become a different operational problem if each retry can produce another external action. Record the conditions under which retrying is allowed and how the reviewer detects an already completed write. The practical decision is whether the workflow reduces total effort while meeting the defined outcome and action limits.

Editorial implications, not an independently measured result.

The comparison

The table scrolls horizontally on smaller screens.

Read alongside Anthropic: Demystifying evals for AI agents.

Your next decision

Which saved artifact and failure case would you inspect before trusting an agent with your own workflow?