Define the cases
Keep inputs and expected behavior together in a versioned dataset.
Compare agent versions on the same cases.
Agent Evaluation. Test candidate responses against a defined rubric, inspect regressions, and record the result for review.
Agent Evaluation. Cases and criteria for the next version.
Compare the same task
Run the baseline and candidate on the same cases and inspect their responses.
Review regressions
Use scores and human judgment to examine where behavior improves or gets worse.
Record the decision
Keep the reviewed comparison with the version and inputs it evaluated.
One failure. A better next version.
Follow a support agent’s unverified refund promise from captured evidence to a reviewed case and a comparison of two releases.
Locate the captured refund promise and the calls around it.
Read what the application reported and distinguish the response from a confirmed refund.
Keep the original request and supply a reviewed expectation: check the policy and request approval.
Inspect the same case on the baseline and candidate, including their different responses.
One candidate result is missing. The comparison remains inconclusive and calls for more evidence.
In practice
The support team compares releases 3 and 4 against refund cases. The candidate requests approval on the disputed case, but another planned result is missing. The team keeps the comparison inconclusive and collects more evidence before making a release decision.
Explore the solutionDiscuss your agent evaluation.
Bring candidate versions and a few representative cases. We’ll discuss the rubric and comparisons needed for your next decision.