We raised $6M in Seed FundingRead more
+
+
+
+
Skip to content

AgentMail EvalsIn collaboration with Scope

How good are LLMs
at using agentic tools?

We put AI agents to work on AgentMail, our own API, to see what makes a tool hard for them.

See the resultsRead the full report

Results

Overall score by model score 0–100

Best harness per model

Same models, other harnesses

11–25
  1. Overall
    94.6
    With docs
    96.7
    Without docs
    92.6
    95% interval
    91.5–97.1
    Succeeded
    97%
    1. Reply-all + BCC90%
    2. Extract confirmation code100%
    3. Reply to password reset95%
    4. Save CSV attachment100%
    5. Summarize one customer’s week100%
    6. Prepare three replies95%
    7. Process signup confirmations100%

    Success 97%Partial 1%Not completed 2%

  2. Overall
    92.3
    With docs
    100.0
    Without docs
    84.5
    95% interval
    90.3–94.3
    Succeeded
    93%
    1. Reply-all + BCC55%
    2. Extract confirmation code100%
    3. Reply to password reset100%
    4. Save CSV attachment100%
    5. Summarize one customer’s week100%
    6. Prepare three replies100%
    7. Process signup confirmations95%

    Success 93%Partial 0%Not completed 7%

  3. Overall
    90.2
    With docs
    88.3
    Without docs
    92.1
    95% interval
    86.6–93.6
    Succeeded
    93%
    1. Reply-all + BCC85%
    2. Extract confirmation code95%
    3. Reply to password reset100%
    4. Save CSV attachment100%
    5. Summarize one customer’s week100%
    6. Prepare three replies95%
    7. Process signup confirmations75%

    Success 93%Partial 4%Not completed 3%

  4. Overall
    88.7
    With docs
    96.9
    Without docs
    80.5
    95% interval
    85.1–91.7
    Succeeded
    90%
    1. Reply-all + BCC90%
    2. Extract confirmation code100%
    3. Reply to password reset100%
    4. Save CSV attachment100%
    5. Summarize one customer’s week95%
    6. Prepare three replies93%
    7. Process signup confirmations50%

    Success 90%Partial 0%Not completed 10%

  5. Overall
    86.4
    With docs
    83.4
    Without docs
    89.4
    95% interval
    82.4–90.1
    Succeeded
    94%
    1. Reply-all + BCC100%
    2. Extract confirmation code100%
    3. Reply to password reset95%
    4. Save CSV attachment100%
    5. Summarize one customer’s week81%
    6. Prepare three replies92%
    7. Process signup confirmations89%

    Success 94%Partial 6%Not completed 0%

  6. Overall
    83.7
    With docs
    81.2
    Without docs
    86.2
    95% interval
    80.4–86.8
    Succeeded
    92%
    1. Reply-all + BCC60%
    2. Extract confirmation code100%
    3. Reply to password reset100%
    4. Save CSV attachment100%
    5. Summarize one customer’s week89%
    6. Prepare three replies100%
    7. Process signup confirmations95%

    Success 92%Partial 4%Not completed 4%

  7. Overall
    79.2
    With docs
    88.2
    Without docs
    70.1
    95% interval
    75.1–83.1
    Succeeded
    84%
    1. Reply-all + BCC85%
    2. Extract confirmation code100%
    3. Reply to password reset100%
    4. Save CSV attachment85%
    5. Summarize one customer’s week95%
    6. Prepare three replies90%
    7. Process signup confirmations35%

    Success 84%Partial 0%Not completed 16%

  8. Overall
    79.0
    With docs
    87.5
    Without docs
    70.6
    95% interval
    75.0–82.8
    Succeeded
    86%
    1. Reply-all + BCC42%
    2. Extract confirmation code95%
    3. Reply to password reset100%
    4. Save CSV attachment95%
    5. Summarize one customer’s week77%
    6. Prepare three replies95%
    7. Process signup confirmations95%

    Success 86%Partial 0%Not completed 14%

  9. Overall
    66.4
    With docs
    72.5
    Without docs
    60.3
    95% interval
    62.6–70.5
    Succeeded
    67%
    1. Reply-all + BCC85%
    2. Extract confirmation code100%
    3. Reply to password reset11%
    4. Save CSV attachment95%
    5. Summarize one customer’s week70%
    6. Prepare three replies100%
    7. Process signup confirmations10%

    Success 67%Partial 0%Not completed 33%

  10. Overall
    65.4
    With docs
    77.6
    Without docs
    53.1
    95% interval
    61.2–69.5
    Succeeded
    67%
    1. Reply-all + BCC90%
    2. Extract confirmation code95%
    3. Reply to password reset11%
    4. Save CSV attachment100%
    5. Summarize one customer’s week60%
    6. Prepare three replies95%
    7. Process signup confirmations15%

    Success 67%Partial 0%Not completed 33%

  11. Overall
    63.5
    With docs
    63.9
    Without docs
    63.2
    95% interval
    57.3–69.6
    Succeeded
    72%
    1. Reply-all + BCC44%
    2. Extract confirmation code84%
    3. Reply to password reset65%
    4. Save CSV attachment79%
    5. Summarize one customer’s week80%
    6. Prepare three replies95%
    7. Process signup confirmations53%

    Success 72%Partial 1%Not completed 27%

  12. Overall
    61.8
    With docs
    75.0
    Without docs
    48.6
    95% interval
    55.7–67.6
    Succeeded
    62%
    1. Reply-all + BCC60%
    2. Extract confirmation code78%
    3. Reply to password reset35%
    4. Save CSV attachment93%
    5. Summarize one customer’s week70%
    6. Prepare three replies65%
    7. Process signup confirmations40%

    Success 62%Partial 6%Not completed 32%

  13. Overall
    61.4
    With docs
    67.6
    Without docs
    55.3
    95% interval
    55.6–67.3
    Succeeded
    65%
    1. Reply-all + BCC80%
    2. Extract confirmation code89%
    3. Reply to password reset32%
    4. Save CSV attachment71%
    5. Summarize one customer’s week80%
    6. Prepare three replies85%
    7. Process signup confirmations20%

    Success 65%Partial 2%Not completed 33%

  14. Overall
    53.3
    With docs
    49.5
    Without docs
    57.0
    95% interval
    48.1–58.5
    Succeeded
    57%
    1. Reply-all + BCC53%
    2. Extract confirmation code80%
    3. Reply to password reset21%
    4. Save CSV attachment75%
    5. Summarize one customer’s week60%
    6. Prepare three replies90%
    7. Process signup confirmations15%

    Success 57%Partial 0%Not completed 43%

  15. Overall
    38.1
    With docs
    45.5
    Without docs
    30.8
    95% interval
    33.7–42.8
    Succeeded
    39%
    1. Reply-all + BCC30%
    2. Extract confirmation code75%
    3. Reply to password reset0%
    4. Save CSV attachment50%
    5. Summarize one customer’s week50%
    6. Prepare three replies70%
    7. Process signup confirmations0%

    Success 39%Partial 2%Not completed 59%

Scores for each model and harness, 0 to 100, from graded runs only.
ModelHarnessOverallWith docsWithout docs
GPT-5.5Codex96.899.594.1
GPT-5.5Hermes94.696.792.6
Grok 4.5Hermes93.092.094.0
Grok 4.5OpenCode92.3100.084.5
Grok 4.5OpenClaw90.288.392.1
GPT-5.6 SolCursor Agent89.698.181.2
GLM 5.2OpenClaw89.390.787.8
GPT-5.5OpenCode88.796.980.5
GPT-6 AstraCodex87.985.490.3
GPT-5.5OpenClaw86.483.489.4
GLM 5.2Hermes83.781.286.2
DeepSeek V4 ProOpenClaw80.982.679.3
GLM 5.2OpenCode79.288.270.1
DeepSeek V4 ProOpenCode79.087.570.6
Claude Sonnet 5Hermes68.971.865.9
Claude Fable 5.1Claude Code67.671.763.5
Claude Opus 4.8Hermes67.278.655.8
Claude Opus 4.8OpenCode66.472.560.3
Claude Sonnet 5OpenCode65.477.653.1
DeepSeek V4 ProHermes63.563.963.2
Claude Opus 4.8OpenClaw61.875.048.6
Claude Opus 4.8Claude Code61.467.655.3
Claude Opus 5.5Claude Code54.566.242.9
Claude Sonnet 5Claude Code53.349.557.0
Claude Sonnet 5OpenClaw38.145.530.8

What we’re testing

Seven everyday jobs, each with a trap any tool can set. To finish, an agent has to learn how the tool behaves instead of guessing.

78%

of runs succeeded

AgentMail leaves spam and mail from unauthenticated senders out of the default inbox view. Agents that don’t ask for them find nothing and decide the email never arrived.

  • Process signup confirmations57%
  • Reply to password reset69%
  • Save CSV attachment91%
  • Extract confirmation code94%
Task specs

Findings

  1. 01

    Agents fail where the tool leaves them guessing.

    A third of failed runs stopped at an inbox that only looked empty. 76% of runs that hit the same thing kept looking and finished.

    Where agents fail
    1. Gave up at an empty inbox35%
    2. Sent the wrong thing22%
    3. Decided it had no access18%
    4. Found it, refused to act17%
    5. Left it half done8%
    6. Our setup, not the agent3%
    7. Not yet categorized6%
  2. 02

    Some models found hidden mail, but hesitated.

    Claude models found it as often as others, but replied to spam-flagged mail in 23% of runs, against 88%.

    Where agents differ
    1. GrokGrok 4.5
    2. GLMGLM 5.2
    3. GPTGPT-5.5, GPT-5.6 Sol, GPT-6 Astra
    4. DeepSeekDeepSeek V4 Pro
    5. ClaudeOpus 4.8, Sonnet 5, Fable 5.1, Opus 5.5
  3. 03

    Harnesses matter.

    Claude Sonnet 5 moved 31 points between harnesses, the software that runs a model. Most models moved about ten or less.

    The harness
    1. Claude Sonnet 5OpenClaw · 38.1Claude Code · 53.3OpenCode · 65.4Hermes · 68.930.8
    2. DeepSeek V4 ProHermes · 63.5OpenCode · 79.0OpenClaw · 80.917.4
    3. GPT-5.5OpenClaw · 86.4OpenCode · 88.7Hermes · 94.6Codex · 96.810.4
    4. GLM 5.2OpenCode · 79.2Hermes · 83.7OpenClaw · 89.310.1
    5. Claude Opus 4.8Claude Code · 61.4OpenClaw · 61.8OpenCode · 66.4Hermes · 67.25.8
    6. Grok 4.5OpenClaw · 90.2OpenCode · 92.3Hermes · 93.02.8
  4. 04

    Docs help, but not everywhere.

    Docs improved six of seven jobs, mostly by showing agents the API existed. On reply-all with a BCC, they made agents worse.

    What the docs do
    1. Summarize one customer’s week
    2. Process signup confirmations
    3. Extract confirmation code
    4. Reply to password reset
    5. Prepare three replies
    6. Save CSV attachment
    7. Reply-all + BCC

How a run is scored

Read the full report

  1. Environment

    A fresh inbox, seeded with mail, and one job described in plain English.

  2. Rollout

    Scope runs each agent until it says the job is done.

  3. Grading

    Finished jobs score higher the faster they ran. Everything else scores 0.

Put your own agent in a real inbox.

Free to start. The same inboxes these agents used.

Start for freeDocs
All systems onlineSOC 2 Compliant

Email Inboxes for AI Agents

support@agentmail.cc

Subscribe to our weekly newsletter.

HomeBuildEnterprisePricingBlogCustomersEvalsCompareUse casesCareersDocsConsoleAgentIDAgent Armor

© 2026 AgentMail, Inc. All rights reserved.

Privacy PolicyTerms of ServiceSOC 2Subprocessors