Why Claude Came Last in Our AI Agent Evals
Claude found hidden email as often as other models but rarely replied to mail flagged as spam. What model defaults mean for AI agent email security.
- evals
- claude
- ai-agents
- agent-security
- spam
- research
TL;DR
- All ten Claude setups finished in the bottom eleven of the 25 model and harness combinations in AgentMail Evals.
- Claude found hidden email as often as the other models. On the two jobs that meant replying to mail flagged as spam, it finished 23% of runs, against 88% for everyone else.
- Claude defaulted to no on flagged mail and the other models defaulted to yes, whether the email was suspicious or the user had approved it.
- If you put an agent on an inbox, write the security rules yourself and enforce them in your email infrastructure.
AgentMail Evals is our benchmark for how well AI agents use a tool they haven't memorized. We ran 25 model and harness combinations through seven everyday email jobs on AgentMail: pull a confirmation code, reply to a password reset, save an attachment, summarize one customer's week. Each run got a fresh inbox and a goal, and we graded what it got done.
Each setup paired a model, such as Claude Sonnet 5 or GPT-5.5, with a harness, the software that runs the model and decides which tools it gets, such as Claude Code, Codex or Hermes. We tested four Claude models across four harnesses, and all ten of those setups ended up near the bottom of the table. Other model families finished 90% of their runs, and Claude finished 63%.
We didn't expect that, so we went back through the runs. Most of the gap comes from a security default. Claude found hidden email as often as everyone else, then held back from replying to mail flagged as spam, including mail the user had told it to handle. The other models mostly went ahead, even on an email a careful person would have questioned. Below are the numbers behind that, and the rules we'd set before putting any model on an inbox.
How did Claude score in AgentMail Evals?
Claude's best result was Sonnet 5 running in Hermes, which placed 15th of 25. Every other model setup scored higher except one, DeepSeek V4 Pro in Hermes. Here are all ten Claude setups.
| Rank of 25 | Model | Harness | Score out of 100 |
|---|---|---|---|
| 15 | Claude Sonnet 5 | Hermes | 68.9 |
| 16 | Claude Fable 5.1 | Claude Code | 67.6 |
| 17 | Claude Opus 4.8 | Hermes | 67.2 |
| 18 | Claude Opus 4.8 | OpenCode | 66.4 |
| 19 | Claude Sonnet 5 | OpenCode | 65.4 |
| 21 | Claude Opus 4.8 | OpenClaw | 61.8 |
| 22 | Claude Opus 4.8 | Claude Code | 61.4 |
| 23 | Claude Opus 5.5 | Claude Code | 54.5 |
| 24 | Claude Sonnet 5 | Claude Code | 53.3 |
| 25 | Claude Sonnet 5 | OpenClaw | 38.1 |
Fable 5.1 and Opus 5.5 ran about three times per task instead of ten, so their scores are less certain than the rest. The full report has every setup, with and without docs.
Did Claude fail to find the hidden emails?
No. Four of the seven jobs hid the email the agent needed, in spam or behind an authentication filter, so the default inbox list came back empty. We tracked what happened next in every run that hit that empty list.
| Model family | Models tested | Job only reads the hidden email | Job replies to the hidden email |
|---|---|---|---|
| Grok | Grok 4.5 | 100% | 96% |
| GLM | GLM 5.2 | 100% | 95% |
| GPT | GPT-5.5, GPT-5.6 Sol, GPT-6 Astra | 98% | 90% |
| DeepSeek | DeepSeek V4 Pro | 88% | 81% |
| Claude | Opus 4.8, Sonnet 5, Fable 5.1, Opus 5.5 | 86% | 24% |
When the job was only to read that hidden email, Claude found it and finished 86% of the time, in line with everyone else. When the job meant replying to it, Claude finished 24%. The drop comes after Claude has already found the email, at the point where it has to act on it.
Which jobs caused the gap?
Two jobs made most of the difference, and both asked the agent to reply to mail flagged as spam. One was a password reset with a reply address that pointed somewhere else, the kind of email a careful person would question. The other was a batch of signup confirmations, where the prompt told the agent: "I've authorized this; don't skip it on principle."
Claude held back on both. Most of the other models went ahead on both, and replied to the suspicious reset almost every time. On those two jobs, Claude finished 23% of runs and everyone else finished 88%. Claude also trailed on the other five jobs, by a smaller margin of 78% to 91%.
Claude's default is no, and everyone else's is yes.
Neither group checked which email was risky and which one the user had approved. Each applied its default to both, so each got one job wrong.
Should an AI agent default to yes or no on suspicious email?
That's your call. A wrong no leaves a job undone until someone comes back to finish it. A wrong yes sends an email you can't unsend, possibly to an address someone else controls. Claude chose the mistake that's easier to clean up, and plenty of teams will want exactly that. Others need the agent to finish the job they asked for.
Neither default makes one model smarter than the other, and neither one is a policy. You can't configure a default or audit it, and it can change with the next model release.
Why do AI models fall back on old patterns with agent tools?
Once we saw this in Claude, we started seeing the same habit in the other models too. About a third of all failed runs listed the inbox, got nothing back, and decided the email had never arrived. One more parameter, include_spam or include_unauthenticated, would have shown it. Other runs found an endpoint called reply-all and trusted it to add a BCC it couldn't add.
Most email code in public repositories only sends mail, and a reply there is a new message with "Re:" in the subject. Our read is that models learned email from that code, so on a tool built for agents they fall back on it.
We read Claude's spam default the same way. Models haven't been trained on agent infrastructure yet, so they apply a blanket rule where a trained agent would read the signal. We think that's good news. A model that can already find the email and do the work only has to learn when to act.
How do you set email security rules for an AI agent?
Until models learn that, don't leave the decision to the model. If you're putting an agent on an inbox and expect it to get work done, write the rules yourself: which senders it can trust, and which actions wait for a person. Put those rules in the infrastructure, where the agent meets them on every call, instead of in a prompt it can weigh against its instincts.
On AgentMail, Lists control which addresses and domains an inbox accepts mail from and sends to, and you can set them per inbox, per pod, or across the organization. A send allowlist would have stopped a reply to the reset email's redirected address before it left. Drafts hold a reply for a person to approve when the agent shouldn't send on its own.
A reply to an unapproved address fails whether GPT-5.5 or Claude wrote it. Once the infrastructure enforces the rules, you can pick a model on how well it does the work.
How could models get better at agent tools?
Agents now work through tools built for them, and those tools show a model things a chat window never does, like a spam label or an authentication result. Knowing how much weight to give those signals is a skill. We suspect it improves the way other skills have, with more training and testing on tools like these.
We'll keep running these jobs as new models and harnesses ship, and the task prompts and grading rules are all in the full report. If you build models or harnesses and want to try them, or think we got something wrong, we'd like to hear from you.
Pick the model for the work, and write the security policy yourself. Every model and harness we tested is in the full results.
