Inbound is the hard part

Most teams building an email agent start with sending. They pick a provider, verify a domain, call one endpoint, and a message lands in someone's inbox. It feels finished. Then the first customer replies, and the agent has to do something with that reply.

That's where the hard part starts. A reply can arrive twice. It can quote three earlier messages and bury the one new line at the top. It can come from an address that looks right and isn't. And to answer it well, the agent has to remember what it said last week, which nothing in the delivery path keeps for you.

None of this is a bug in any one product. It comes from how email was designed, decades before anyone wanted software to hold a conversation over it. Below are the four places inbound breaks, what each one costs an agent, and the design that removes most of the work.

Why sending email is the solved half

Sending is solved because the hard parts were turned into a service years ago. Deliverability, DNS setup and bounce handling are things a provider runs for you. Your code makes one call and gets back an ID. If that call times out, an idempotency key makes the retry return the original message. No second email goes out.

AgentMail's send endpoints work that way. A repeated key returns the first message ID and thread ID. The same key on a different request gets a 409 error. Mistakes are loud. The idempotency docs cover the key rules.

Receiving runs the other direction. Mail arrives when someone else decides to send it, in whatever shape their client produced, and your agent has no say in the format. Every guarantee you had on the way out is missing on the way in.

Why does an agent get the same email twice

An agent gets the same email twice because the delivery path retries whenever an acknowledgment is late. That rule is written into the protocol. RFC 5321, the standard for SMTP, tells receiving servers they must answer quickly "to avoid receiving duplicate messages as the result of timeouts." Duplicates are part of the design.

Webhooks inherit the same behavior one layer up. If your endpoint is slow to answer, the sender assumes it failed and tries again. Suppose the handler waits on a model call before it returns. The retry starts the same job a second time, and the agent answers twice, books two meetings, or refunds one order twice.

Two fixes go together. Return 200 right away and do the work in a background worker. Record an ID for each delivery so a retry gets recognized and dropped. AgentMail delivers webhooks through Svix, and the svix-id header stays the same across retries of one message, which makes it the natural key. Verify the signature first, as the webhook verification docs show, then run a handler like this one.

# Acknowledge fast, drop retries by svix-id, hand the new text to a worker
import queue
from flask import Flask, request

app = Flask(__name__)
jobs = queue.Queue()
seen_ids = set()  # use a table with a unique key in production

@app.route("/webhooks", methods=["POST"])
def handle_webhook():
    delivery_id = request.headers["svix-id"]
    if delivery_id in seen_ids:
        return "OK", 200  # a retry of a delivery already taken
    seen_ids.add(delivery_id)

    payload = request.json
    if payload["event_type"] == "message.received":
        message = payload["message"]
        new_text = message.get("extracted_text") or message.get("text")
        jobs.put((message["thread_id"], message["message_id"], new_text))

    return "OK", 200

# Posting the same delivery twice prints: 200 1, then 200 1

The set lives in memory here so the example stays short. In production the seen IDs go in a table with a unique constraint. Otherwise a restart forgets them.

How does an agent know which conversation a reply belongs to

A reply belongs to a conversation only because of three headers the sender's client may or may not set. Message-ID names each message, In-Reply-To points at the message being answered, and References lists the chain before it. RFC 5322 defines all three, and nothing enforces them. A reply from a phone and a reply from a forwarding rule can carry different chains for the same conversation.

The body makes it worse. Most replies quote the whole history under the new line, plus a signature and a "Sent from my iPhone" footer. An agent that reads the raw text pays for that history on every turn. Worse, it can mistake an old request for a new one.

In AgentMail, threading is automatic from those three headers. Every message also carries an extracted_text field with the quoted history and boilerplate removed, which is what the handler above reads first. The harder cases, like changed subjects and forwarded chains, are in email threading for AI agents.

Who is actually writing to your agent

The From line on an email proves nothing by itself. RFC 5321 calls SMTP mail "inherently insecure." It adds that a casual user can create messages that "trick a naive recipient into believing that they came from somewhere else." SPF, DKIM and DMARC, the DNS records that let a receiving server check a message really came from the domain it claims, were added later. They only help when the sending domain publishes them.

For a person, a spoofed sender is a phishing risk. For an agent that can issue refunds, change a shipping address or share a file, it's an instruction from a stranger. The agent has to know whether a message was authenticated before it acts. It also needs a short list of who may ask for what.

AgentMail splits these cases at the event level. Mail that arrives without authentication headers comes in as message.received.unauthenticated, and spam and block list matches get their own event types. None of the three reaches your webhook unless you subscribe to it by name. Lists allow or block senders per inbox, pod or organization, with a separate pair of lists for replies to mail the agent sent.

What does an agent need to remember about past email

An agent needs the whole conversation, not the latest message. A customer who writes "yes, the second option" is answering something the agent said two days ago. SMTP delivers a message and forgets it. A webhook does the same. Once your endpoint returns 200, the event is gone unless you stored it.

So teams that only consume events end up building a store. They save every inbound and outbound message and index it by thread. They track which ones got an answer. Then they keep it all consistent when a retry or a forward arrives out of order. That store is a mailbox. They built one without meaning to.

Suppose the agent handles three customers at once. Each one needs its own history, its own record of what was promised, and its own state, like waiting on the customer or escalated. When threads are a stored object, the agent fetches the thread by ID and reads every message in order. Labels such as unreplied and escalated carry the state.

What a mailbox gives you that an event stream does not

A mailbox answers each of the four gaps with stored state, where an event stream only says that something arrived. The table lines up what the protocol hands you against what an agent needs before it can act.

The agent's job is to decide what to say. Everything in the right column is plumbing it shouldn't rebuild for each new inbox. That's the case for a mailbox. The state lives with the email, and the event becomes a notice that something changed.

To wire that notice up, trigger an AI agent when an email arrives walks through the handler. Testing inbound email webhooks covers replaying duplicates and slow responses before production. For a side by side of inbound providers, read the best inbound email APIs for AI agents.

What AgentMail does not do

AgentMail retries webhooks, so your endpoint still has to dedupe on the delivery ID and return fast. It doesn't make your handler idempotent for you. It also doesn't decide who your agent should trust. Authentication events and lists narrow the field, but the rule for which sender can ask for a refund is yours to write. When a message has no extracted text, the agent should fall back to the full body.