"It worked on the demo" is not an evaluation. Production agents need continuous, automated evals across correctness, cost, and latency.
What to measure
- Task success rate on a golden set of representative inputs.
- Trajectory quality — did the agent take a reasonable number of steps, or loop forever?
- Tool-call accuracy — right tool, right arguments.
- Cost per resolution — tokens × price + tool spend.
- Latency P50 / P95.
Tooling
- LangSmith — traces, datasets, evaluators.
- Langfuse — open-source tracing and evals.
- Braintrust — eval-driven development.
- Arize Phoenix — open-source LLM observability.
- OpenAI Evals — framework and shared eval registry.
Workflow
- Log every trace from day one.
- Curate 50–200 real examples into a golden set.
- Write LLM-as-judge or code-based graders per example.
- Run evals on every prompt or model change; block deploys on regressions.
See Hamel Husain's Your AI product needs evals for the canonical playbook.