What does a large gap between pass@1 and pass@5 tell you about an agent?

University of Michigan - Ann Arbor

Trajectory evaluation scores the path rather than the endpoint: were the right tools chosen, were failures recovered from, was the step count reasonable. It is what surfaces agents that reach the right answer by an unrepeatable accident.

Trajectory Evaluation

Your agent's outcome score improved after a prompt change, but users report it feels slower and more expensive. Design an evaluation programme that would have caught this before release, and explain what each component measures.

Learn Before

Related