AI Agent Evals

AI agents do more than answer questions. They issue refunds, update records, and hand work to other agents, and a transcript that looks perfect can hide a refund that happened twice.

AI Agent Evals is a practical field guide to evaluating agents by what they changed in the systems they touch. Its central idea, the Evidence Ladder, gives product managers and AI engineers one rule they can apply in any design review: an agent should act on its own only up to the highest level of proof you can verify automatically.

The book follows one agent from start to finish: Speedy, the customer support assistant at Pronto, a fictional online grocery store. Every chapter tests one of Pronto's real policies, from the $50 refund rule to substitution opt-ins and safety escalations.

You'll learn how to:

• Write task contracts that define success, including correct refusals and escalations
• Find what breaks with error analysis, before choosing any metric
• Build LLM judges, then grade them against expert labels
• Grade tool calls and trajectories without dictating the path
• Red-team boundaries, permissions, guardrails, and data leakage
• Evaluate RAG, memory, handoffs, and multi-agent systems
• Monitor production, catch drift, and gate releases in CI/CD
• Choose between large and small models, and route between them, on evidence
• Promote an agent's autonomy only as far as the evidence supports
• Make the release call, ship, limit, or hold, with a gate table written before the run

Every example is runnable Python, and a companion repository on GitHub runs the whole book end to end.