Evaluating how AI agents use tools

1The question

AI agents increasingly do their work through tools: APIs, internal services, products built for them or adapted to them. Whether an agent gets the job done depends as much on the tool as on the model. This report asks what makes a tool hard for an agent to use, and what separates the agents that cope from the ones that don't.

We build AgentMail, an email API for AI agents, and used it as the test bed. Most email code in public repositories only sends mail, through libraries like smtplib, SendGrid, or Resend: a reply is just a new message with Re: in the subject, and keeping customers apart is left to the application. AgentMail is built around what an agent needs that send-only APIs don't provide:

• Threads are server-side objects.
• A reply is a call on a specific message_id.
• A draft carries a pointer to the message it answers.
• Spam and unauthenticated mail stay out of list results unless the caller asks for them.
• Customers are isolated in pods, each with its own scoped reads.

Each of these is documented, and each has a familiar alternative that looks like it should work. So every task asks one question:

Given a goal and the docs, does an agent get the job done on a tool it hasn't memorized?

AgentMail stands in for agentic tools in general. Every trap these tasks set has a counterpart in other APIs:

We ran 25 agents, each a model inside a harness such as Claude Code or Codex, through seven of these tasks, with and without links to the docs. Each run gets its own inbox, seeded through AgentMail's real inbound pipeline so the spam labels are real, and goes until the agent says it's done (Figure 1). Our partners at Scope ran every agent; we wrote the tasks and graded the recordings. Section 6 has the details.

2Where agents fail

Agents rarely fail for lack of skill with a tool. They fail where the tool leaves something unsaid and they have to judge it on their own.

80% of runs finished, and writing the replies, files and summaries was almost never what went wrong. The failed runs cluster in a few places (Figure 2), and nearly every one is a place where the tool left the agent guessing:

• A filtered list that looked empty. The agent concluded the email had never arrived.
• An endpoint that did less than its name. The agent trusted the name.
• A tool it couldn't see. The agent decided it had no access, sometimes without trying.
• A spam label against the user's word. The agent refused a job it had been asked to do.

Something hidden by default

AgentMail's default inbox list leaves out two kinds of mail: messages flagged as spam, and messages from senders that couldn't be authenticated. Both are one parameter away, and the docs explain how to ask for them. Plenty of tools work this way: a search API returns the first page, a database query skips soft-deleted rows, a dashboard hides archived items. The answer looks complete, and nothing in it says otherwise.

This was the most common place to fail: about a third of failed runs ended here. The agent listed the inbox, saw nothing, and concluded the email had never arrived. Figure 3 is a typical run. Claude Opus 5.5 in Claude Code checked the inbox eight different ways, including spam, trash and junk, but never asked for unauthenticated mail, the one view the documented path uses.

An empty list wasn't fatal on its own: 76% of runs that hit one kept looking and finished. The runs that failed are the ones that stopped there.

Treat an empty result as a question, not an answer.

That holds for agents and for the tools they use. A tool can answer the question before it's asked: a list that says “2 messages filtered as spam” leaves nothing to guess.

An operation that only looks right

Replying to everyone with a manager on BCC takes AgentMail's ordinary reply endpoint, with reply_all on and the manager in bcc. AgentMail also has a reply-all endpoint whose name matches the job exactly, but it can't add anyone new.

Agents that read the docs found that endpoint and trusted its name. It's the one job where linking the docs made agents worse: success fell from 81% to 62%. Claude Opus 4.8 got it right every time without docs and 48% of the time with them (Figure 4).

An endpoint's name is a promise, and agents rely on it the way a person skimming the docs would. The same thing happened elsewhere: a draft that went out as a real email, a reply posted to the wrong conversation. The fix is the tool's: name operations for what they do, and say plainly what they don't.

Signals that point both ways

Some runs found the right email and refused to act on it because it was flagged as spam or looked suspicious. Nearly all of these refusals came on two jobs, and they point in opposite directions. The password-reset email carries a spam label and a reply address that redirects elsewhere; pausing there is what a careful person would do. The signup confirmations come from a user who says real signups land in spam and asks the agent to process them; refusing there ignores a direct instruction.

Either way, the agent had to weigh the tool's warning against the user's request with nothing to go on. A tool can help by saying what its labels mean: whether “spam” is a verdict or a guess, and how much it should count when the user has asked for the message.

3Where agents differ

Most of the difference between agents comes from one model family on one kind of job. Other models finished 90% of their runs; Claude models finished 63%. On five of the seven jobs the gap is modest (78% against 91%). Most of it sits in the two jobs that ask for a reply to mail flagged as spam, the password reset and the signup confirmations: Claude models finished 23% of those, everyone else 88%.

It isn't that Claude models couldn't find hidden mail. Among runs that hit an empty inbox, they recovered 86% of the time when the job was only to read the mail, close to other models (88–100%), but 24% when the job meant replying to it (81–96%).

The likeliest reason is caution: the same models that refused suspicious mail also searched less for it. It's a reasonable instinct applied too broadly, and one a tool can steer by saying which of its signals matter.

The harness

An agent is a model plus a harness, the software around it (Claude Code, Codex) that decides which tools the model gets, what it sees and when it stops. So we score every model-and-harness pair separately and never average a model across harnesses.¹

For most models the harness moved the score by 10 points or less (Figure 6). For Claude Sonnet 5 it moved 31: 69 in Hermes, 38 in OpenClaw, where 75% of its failed runs without docs stopped because it decided it couldn't reach AgentMail, some before making a single request.

Comparing models fairly. OpenCode, Hermes and OpenClaw ran the same six models, so they give the cleanest comparison: Grok 4.5 leads in OpenCode and OpenClaw, GPT-5.5 leads in Hermes. A gap that holds in every harness belongs to the model; one that changes with the harness belongs to the harness.

4What the docs do

Every task ran twice, and nothing else changed. With docs, the prompt opens with links to the AgentMail docs pages that explain the right method. Without docs, there are no links, though an agent can still look the docs up on its own.

Linking the docs lifted success from 76% to 84% and helped on six of the seven jobs, mostly by telling agents the tool was there. Without links, 6% of runs decided there was no way to reach AgentMail and stopped; with them, 1% did.

Past that, the docs mattered less: 24% of agents scored higher without them, and on reply-all they pointed agents to the look-alike endpoint (Figure 7).

5What it means

What makes a tool hard for an agent isn't how complex it is. It's how much the tool leaves the agent to infer.

Every failure here points to something a tool, a harness or a model can do better. The biggest lever is the tool's, because every agent goes through it.

For tool builders. Say what a list filtered out. Name operations for what they do, and document what they don't. Make the tool easy to find; here, the docs' biggest effect was telling agents the API existed. And say how much a label like spam should count against what the user asked for.

For harness builders. Make sure the model can find the tool it's meant to use, and make checking the default: read back every write, and treat an empty result as a question.

For model builders. Caution with suspicious content is right, but it shouldn't override an explicit instruction or stop a search before it starts.

What we're changing in AgentMail

Every failure here is also a lesson for us. These are the ones we're taking forward, roughly in order of how many failed runs they would have saved:

1. Say what a list leaves out. About a third of failed runs (35%) stopped at an inbox that only looked empty. The likely fix is for list responses to say how many messages were held back as spam or unauthenticated, so an empty result reads as “filtered”, not “nothing here”.
2. Say what each endpoint can't do. The reply-all endpoint did what it was built for, but its name promised more. Its docs should say plainly that it can't add recipients and point to the reply endpoint for a BCC. We expect to find other endpoints like it once we look.
3. Make the API hard to miss. Without links to the docs, 6% of runs decided they couldn't reach AgentMail at all. We want our docs, llms.txt and setup instructions easier to find from wherever an agent starts, so none gives up before its first request.
4. Give labels a meaning. “Spam” and “unauthenticated” are a classifier's judgement, not a verdict. Documenting what they mean, and how to weigh them when a user has asked for the message, should help both the agents that refused too readily and the ones that should have paused.
5. Keep testing the way agents work. These seven tasks are a cheap check. We'd like to run them against changes to our API and docs, so the problems an agent would hit show up before they ship, and to rerun this eval once the fixes above land.

6How we tested

Each task is modeled on something agents really do on AgentMail: confirm a signup, answer a password reset, save a file from a message, summarize one customer's week, draft replies for a person to approve. And each can be done two ways. The documented way works, and it's in the docs. The familiar way is what most public email code does: it looks right, but here it fails.

Prompts state the goal and nothing else. They never name a feature or parameter, because a prompt that said "use include_spam" would only test whether the agent follows instructions. In four tasks, the email the agent needs is hidden from the default inbox view (Figure 8).

One task in full

Here is the hardest task, as the agent sees it and as we grade it. The other six follow the same shape.

How a run is scored

We grade each run on what it actually did: the requests it made, and the replies, files and answers it produced. Most checks are deterministic, like whether a code matches or a reply landed in the right thread. A few need a person to read the evidence, like whether a summary sticks to the right customer. And we grade the result, not the route: requiring "the right method" would only measure agreement with us.² Every run ends in one of three outcomes:

Three rules settle the edge cases:

• Claims need proof. An agent's account of its work counts only where its recorded requests back it up. "Done," or a harness reporting success, counts for nothing.
• Any working route counts. A different API path to the same outcome is a success, and so is a run that hit an error and recovered. A reply the API accepted counts as sent.
• Contradictions fail the run. A reply to a decoy, another customer's data in a summary, or a forward in place of a BCC.

A model-and-harness pair's score averages its runs on each task, then weighs the seven tasks equally, and runs with and without docs equally. Runs we couldn't grade from the evidence are left out. Every score has a 95% confidence interval: about ±2 to ±6 points for most pairs, and up to ±8 for the three that ran fewer times.³

7Full results

#	Model and harness	Overall	95% interval	With docs	Without docs	Outcomes
01	GPT-5.5 Codex	96.8	94.4–98.9	99.5	94.1	136 success, 3 partial, 0 not completed
02	GPT-5.5 Hermes	94.6	91.5–97.1	96.7	92.6	134 success, 1 partial, 3 not completed
03	Grok 4.5 Hermes	93.0	90.3–95.4	92.0	94.0	133 success, 0 partial, 5 not completed
04	Grok 4.5 OpenCode	92.3	90.3–94.3	100.0	84.5	130 success, 0 partial, 10 not completed
05	Grok 4.5 OpenClaw	90.2	86.6–93.6	88.3	92.1	127 success, 6 partial, 4 not completed
06	GPT-5.6 Sol Cursor Agent	89.6	86.0–93.2	98.1	81.2	126 success, 1 partial, 12 not completed
07	GLM 5.2 OpenClaw	89.3	84.5–93.6	90.7	87.8	95 success, 2 partial, 6 not completed
08	GPT-5.5 OpenCode	88.7	85.1–91.7	96.9	80.5	121 success, 0 partial, 14 not completed
09	GPT-6 Astra Codex	87.9	80.7–95.0	85.4	90.3	37 success, 0 partial, 5 not completed
10	GPT-5.5 OpenClaw	86.4	82.4–90.1	83.4	89.4	117 success, 7 partial, 0 not completed
11	GLM 5.2 Hermes	83.7	80.4–86.8	81.2	86.2	123 success, 5 partial, 6 not completed
12	DeepSeek V4 Pro OpenClaw	80.9	76.7–84.8	82.6	79.3	114 success, 1 partial, 15 not completed
13	GLM 5.2 OpenCode	79.2	75.1–83.1	88.2	70.1	118 success, 0 partial, 22 not completed
14	DeepSeek V4 Pro OpenCode	79.0	75.0–82.8	87.5	70.6	111 success, 0 partial, 18 not completed
15	Claude Sonnet 5 Hermes	68.9	63.0–74.5	71.8	65.9	97 success, 8 partial, 26 not completed
16	Claude Fable 5.1 Claude Code	67.6	59.6–75.8	71.7	63.5	30 success, 0 partial, 11 not completed
17	Claude Opus 4.8 Hermes	67.2	61.6–72.7	78.6	55.8	95 success, 4 partial, 36 not completed
18	Claude Opus 4.8 OpenCode	66.4	62.6–70.5	72.5	60.3	93 success, 0 partial, 45 not completed
19	Claude Sonnet 5 OpenCode	65.4	61.2–69.5	77.6	53.1	93 success, 0 partial, 46 not completed
20	DeepSeek V4 Pro Hermes	63.5	57.3–69.6	63.9	63.2	96 success, 2 partial, 36 not completed
21	Claude Opus 4.8 OpenClaw	61.8	55.7–67.6	75.0	48.6	82 success, 8 partial, 43 not completed
22	Claude Opus 4.8 Claude Code	61.4	55.6–67.3	67.6	55.3	87 success, 3 partial, 44 not completed
23	Claude Opus 5.5 Claude Code	54.5	47.4–61.6	66.2	42.9	23 success, 1 partial, 18 not completed
24	Claude Sonnet 5 Claude Code	53.3	48.1–58.5	49.5	57.0	78 success, 0 partial, 60 not completed
25	Claude Sonnet 5 OpenClaw	38.1	33.7–42.8	45.5	30.8	54 success, 3 partial, 81 not completed

Scores are 0–100, from graded runs only, with 95% intervals. Outcome bars, left to right: success, partial, not completed. Explore the interactive chart