An AI Agent Tested 3 Email Vendors and Caught a Hidden Failure

SaaStr, a software business community, used an AI agent to compare three email delivery vendors across more than 500,000 emails in one week. The agent, called 10K and described as SaaStr's AI VP of Marketing, ran the whole test. It also found a send that showed "Completed" on the dashboard while no emails actually went out.

What happened

Vendor tests used to be simple. A few demos, a two-week trial, a few hundred test emails and a spreadsheet. SaaStr says the choice often came down to which sales rep replied fastest and what the team already knew how to use.

This time the agent ran the comparison at real volume. Its cleanest test split one campaign into two matched groups of about 2,300 people each. The content was identical and both went out the same morning. One group used the current email sender. The other used a challenger.

The challenger got 2.4 times the opens and 2.9 times the clicks.

The failure behind the green light

On one daily send, the current vendor's workflow said "Completed." The agent looked at the detailed step-by-step records underneath that label. It found 80,708 attempts and 80,708 errors. Zero emails were sent. The vendor's status page showed green the whole time.

The agent then ran four test sends, changing one thing each time. All four failed the same way. That ruled out SaaStr's content, setup and email list, and pointed to the vendor's own delivery system. SaaStr notes a person in a two-week trial would likely have seen "Completed" and moved on.

How the vendors were scored

The agent would not pick a winner on opens alone. Its scorecard covered three things:

• Whether emails were truly delivered, confirmed at the source
• Engagement from real people
• Operator effort, meaning how much staff time each vendor takes

SaaStr says it had never put staff effort in a vendor comparison before. It turned out to be one of the biggest differences between the vendors.

What to keep in mind

The agent listed its own limits. The split test was one campaign with about 2,300 people per side. Open counts are noisy. Raw click counts include security scanners, which are tools that open links automatically. Still, clicks moved the same way as opens, which made the gap hard to explain away.

The results also come from SaaStr itself, and the vendors are not named in the material.

What it means for buyers

SaaStr has three people and no email deliverability team. It says a test this deep would normally mean hiring a consultant for a quarter. The agent finished in a few days alongside its other work.

For anyone buying software, SaaStr's advice is to have an agent run the next vendor test and hand it the scorecard up front. For sellers, the message is that customers can now test them at volume, every week.