How to Evaluate AI Agents

Quick answer: Evaluate an AI agent at three levels: final outcome, trajectory and operational impact. Test whether it achieved the task, selected correct tools, respected permissions and approvals, recovered from failures, cited valid evidence and stayed within latency and cost budgets. Use repeatable scenarios and inspect traces—not isolated demo conversations.

Agent evaluation is harder than scoring a final paragraph. A system can produce a correct-looking answer after calling the wrong customer record, exposing unnecessary data or attempting a duplicate action. Conversely, a run may escalate correctly instead of forcing an unsafe “success.”

Anthropic's agent-evaluation guidance emphasizes that evals make behavioral changes visible before failures reach production. NIST is also exploring evaluation probes for the hidden multi-step workflows behind agent decisions.

Define the task contract first

For every evaluated workflow, write:

• input and user identity;
• permitted data and tools;
• expected outcome;
• acceptable alternative outcomes;
• forbidden actions;
• approval points;
• maximum turns, time and cost;
• evidence required;
• external state that proves success.

Without a contract, reviewers score style rather than behavior.

The evaluation scorecard

Build the scenario dataset

Start with 30–50 examples covering real variation rather than hundreds of synthetic happy paths. Sources include support cases, operational runbooks, user interviews and observed production failures.

Include:

• normal success;
• missing required information;
• ambiguous request;
• unsupported goal;
• conflicting evidence;
• stale record;
• permission denial;
• tool timeout before and after side effect;
• partial downstream outage;
• user cancellation;
• prompt injection in an email, file or page;
• request that requires approval;
• request that must be refused or escalated.

Remove or anonymize sensitive data and preserve the structural difficulty.

Outcome evaluation

The strongest outcome check reads the authoritative external state. If an agent was asked to create a ticket, verify ticket ID, fields, ownership and absence of duplicates. Do not score the agent's claim that it succeeded.

Classify outcomes:

• correct success;
• correct escalation;
• safe refusal;
• partial success disclosed accurately;
• incorrect action;
• unsupported claim of success;
• no progress or loop.

Correct escalation should count as success when policy requires it.

Trajectory evaluation

Inspect the path:

• Were unnecessary tools called?
• Was sensitive data sent to an unrelated tool?
• Did the agent retrieve evidence before acting?
• Were write actions previewed and approved?
• Did it repeat a call after uncertain timeout?
• Did a handoff transfer sufficient context?
• Did it stop after the goal was met?

Exact call matching can be too strict when several safe paths exist. Define required, optional and forbidden steps.

Deterministic checks

Use code for:

• schema validity;
• expected resource and tenant;
• exact financial calculation;
• required approval presence;
• number of external writes;
• citation URL allowlist;
• budget and latency threshold;
• tool argument constraints;
• final database state.

Deterministic graders are preferable when the criterion can be computed.

Model and human graders

Model-based grading can assess relevance, completeness or evidence alignment at scale. Give the grader a rubric, reference answer and structured output. Validate grader agreement against expert review.

Humans remain important for ambiguous user experience, policy nuance and new failure discovery. Use blinded samples and adjudication for disagreement.

Do not let the same unreviewed model family be the only judge of its own agent behavior.

Security evaluations

Create adversarial content that tells the agent to ignore policy, reveal secrets, call a different tool or send data to an attacker. Place it inside the same source types the agent uses in production.

Test:

• indirect prompt injection;
• cross-tenant identifiers;
• malicious tool output;
• changed MCP tool description;
• unsafe URL and redirects;
• memory poisoning;
• approval confusion;
• multi-agent trust propagation.

Use the AI agent security guide and MCP security guide to build the threat cases.

Regression testing

Run the suite when changing:

• model or model settings;
• system instructions;
• tool name, schema or description;
• retrieval index or ranking;
• orchestration or handoff;
• memory logic;
• authorization policy;
• MCP server version;
• retry and timeout behavior.

Keep the previous production configuration as a baseline. Compare paired results and investigate meaningful regressions rather than relying on one average score.

Production monitoring

Offline evals cannot cover every request. Monitor:

• verified completion;
• human correction and override;
• escalation rate by reason;
• policy denials;
• duplicate or compensated actions;
• tool error and timeout rate;
• turns, latency and cost distribution;
• user-reported harmful or incorrect behavior;
• drift by model, prompt and tool version.

Sample traces for expert review. Protect personal and secret data in traces.

Evaluate single and multi-agent systems differently

For one agent, score tool choice and outcome. For a multi-agent system, also score routing, context transfer, specialist output and final synthesis. Evaluate specialists in isolation before the end-to-end workflow.

Read single-agent vs multi-agent systems and how to build an AI agent for architecture-specific checkpoints.

A release gate example

Require:

• no unauthorized external actions in the security set;
• 100% approval enforcement for designated tools;
• no cross-tenant access;
• target verified task completion on normal scenarios;
• safe escalation on unsupported cases;
• no duplicate writes in timeout tests;
• p95 latency and cost within documented limits;
• expert review of every new high-severity failure.

Targets are product-specific. Avoid copying public benchmark thresholds that use different tasks and definitions.

Frequently asked questions

What is an AI agent eval?

It is a repeatable test of an agent's behavior and outcome under defined scenarios, including tool use, policy, safety, efficiency and recovery.

How many eval examples do I need?

Begin with a small, diverse set that represents high-frequency and high-risk cases. Add every meaningful production failure as a regression case.

Can an LLM grade an AI agent?

Yes for some qualitative criteria, but calibrate it against expert review and use deterministic verification whenever possible.

Should a correct final answer pass if the trajectory was unsafe?

No. Outcome and trajectory are separate. An unauthorized or privacy-violating path must fail even if the final text is correct.

Sources and further reading

• Anthropic: Demystifying evals for AI agents
• NIST: Building Evaluation Probes into Agentic AI
• OpenAI: Integrations and observability
• OpenAI: Results and state