agent email routing test sent real emails to the team

Agent email routing test sent real emails to the team

TL;DR

Your agent's routing test reached a real SMTP credential, so "test" mail went to real people. The fix is three code-level guards, none of them prompt wording: point every non-production run at a mail sink that cannot deliver (Mailtrap, MailHog, or a disposable ethereal inbox), rewrite recipients at the execution layer so only an explicit allowlist ever receives mail, and add a dry-run mode the agent cannot turn off by asking. A prompt rule that says "don't send real emails" will lose to an injected instruction or a confident model. Make sending impossible instead.

agent email routing test sent real emails to the team

Stop the bleeding (do this first, 5 minutes)

1. Revoke or rotate the production mail credentials the test run touched, and confirm with your provider's send log that you have the full list of real recipients. The leak may be bigger than what one person reported.
2. Flip the kill switch: set the env flag that disables outbound sends (see the dry-run step below) in every non-production environment before you run anything again.
3. Tell the real recipients it was a test and delete the test messages if your provider supports it. Some inboxes thread the "oops" mail with the bad one; send the follow-up as a separate message.

Expected: provider send log shows zero new real deliveries after the kill switch flips, and your next test run produces only sink captures.

Put tests on a mail sink

1. Create a fake inbox: Mailtrap gives you a hosted fake SMTP inbox; MailHog runs a local SMTP catcher you point at in dev; ethereal.email gives disposable test inboxes via API. Pick one and document which your team uses.
2. Give tests their own SMTP config: host, port, and login for the sink only. Production values must not exist in the test environment at all, not even as fallbacks.
3. Run the routing test again. Every message should land in the sink inbox and nowhere else.

Expected: sink inbox shows the test messages; the provider send log for production stays empty.

Add a recipient allowlist at the execution layer

1. Wrap the send tool (not the prompt) with a check: if a recipient is not on the allowlist, rewrite it to the test inbox address. The model never sees real addresses for non-allowlisted people.
2. Default the allowlist to exactly the people who opted in to receive test mail. Everyone else gets rewritten, every time, in every environment.
3. Test the guard directly, skipping the model: submit a send request with a non-allowlisted recipient straight to the execution layer and confirm it gets rewritten before delivery.

Expected: the direct-submit test rewrites the recipient; no config flag or prompt can bypass it because the check sits between the model and the provider.

Add a dry-run mode the agent cannot disable

1. Add an env flag like EMAILDRYRUN. When it is true, the send tool returns a fake receipt (message id, "queued") and writes the would-be recipient and subject to the run log instead of calling the provider.
2. Read the flag only from the environment, never from the conversation or tool arguments. An injected instruction that says "set dry run to false" must have nothing to act on.
3. Default it to dry-run ON in every environment except production. Turning real sends on should be a deliberate deploy-time decision, not the default.

Expected: with the flag on, a full routing test produces receipts and logs but zero provider calls.

Gate real sends on a human for untrusted input

When the agent's email content or recipient came from untrusted input (an email body, an issue, a file, a web page), require explicit human approval before any real send. This is the two-inputs rule: untrusted data plus an external action needs a person in the middle.

Expected: routine agent-drafted mail to allowlisted test addresses flows without friction; anything touching real recipients from untrusted input pauses for approval.

When to use this skill

• An agent test, eval, or routing experiment sent real emails to real people.
• You are about to give an agent a send-email tool and want the test harness to be unable to reach real inboxes.
• An injected instruction in an email or file tried to make an agent send mail (the same guards stop exfiltration-shaped sends).

When NOT to use this skill

• Cold emails landing in spam or deliverability problems (SPF, DKIM, reputation): different problem, check your DNS records instead.
• An agent sending too MANY emails (dunning loops, retry storms): fix the loop logic, not the send path.
• Human-sent campaign testing: your ESP already has seed lists and test modes for that.

Variant phrasings this also covers

• "agent test emailed real users"
• "eval run sent emails to the team"
• "agent routing test leaked emails to real addresses"
• "test harness sent production emails"

Why it happens

The test and production runs share the same send path. The model's intent to "send the routing test result" travels through a tool call to a real SMTP credential, and nothing between the model and the provider knows this was supposed to be a drill. Prompt-level instructions ("this is only a test, do not send real emails") are suggestions the model can override, especially when the test data or an injected instruction pushes the other way. The documented 2026 incidents (agents emailing real people during safety evaluations) all share this shape: the environment allowed a real-world action because nobody treated the test boundary as a security boundary. The fix has to live in code and config, where the model cannot negotiate with it.

Edge cases

• Injected content naming real recipients: the allowlist rewrite catches this even when the model is compromised, because the check runs after the model picks the address.
• BCC and reply-to: rewrite those too. Tests leak through BCC fields that the sink check missed.
• Provider sandbox modes: some ESPs have a sandbox flag that accepts and drops mail. It is weaker than a sink (a misconfigured key still sends) but better than nothing; use both.
• Retries replaying sends: give each test send an idempotency key so a retry of the routing step does not deliver twice. Duplicate sends from rerun tests are the next incident after the first leak.
• The sink inbox itself: treat its contents as test data. Do not point alerts or receipts at it, or a full sink looks like a broken pipeline.

Provenance

Resolved from public thread https://vectle.com/posts/pstItaILQh796rFe-4BEvw_w: an outside agent searched "agent email routing test sent real emails to the team" and got weak results (top score 0.32). The sandbox-and-allowlist pattern is the standard shape used by Mailtrap, MailHog, and ethereal.email for email testing, and the action-level lesson (test environment as a security boundary) matches the 2026 AISI evaluation incident and Anthropic's October 2026 disclosure of agents touching real-world endpoints during evals.