How Many Cold Emails Do You Need for a Valid A/B Test?

How many cold emails do you need to send before an A/B test result means anything?

Usually far more than most teams send. To tell a 3 percent reply rate from a 5 percent reply rate with standard confidence, you need roughly 1,500 contacts in each version. Smaller differences need many more. This tutorial shows you how to calculate the number yourself in a Google Sheet before you launch the test.

Many outbound tests end too early. A team sends a few hundred emails per version, sees one version get a handful more replies, and declares a winner. The next month, the "winner" performs the same as the old version. Nothing was learned, because the test never had enough volume to separate signal from noise.

I build outbound and reporting systems for B2B SaaS teams using Clay, Apollo, HubSpot, and sequencing tools. Calculating the sample size before a test is one of the cheapest habits you can add. It takes ten minutes and saves weeks of chasing results that were never real.

What is a sample size, and why does it matter for outbound tests?

A sample size is the number of contacts each version of your email needs to reach before the difference in results is likely real rather than random. Reply rates in cold email are low, so random swings are large compared with the effect you are trying to measure. Without enough volume, chance decides your winner.

Think about flipping a coin ten times. Getting seven heads is not strange, and it does not mean the coin is weighted. Cold email works the same way. With a few hundred sends, one version can easily get a few extra replies by luck. The smaller the reply rate, the more volume you need to see past that luck.

This is why I calculate the number before the test, not after. Deciding the sample size up front stops you from peeking at results every day and stopping the moment one version looks ahead. That habit, often called peeking, is one of the most common ways teams fool themselves.

What four inputs do you need before you calculate?

You need your baseline reply rate, the smallest improvement you care about detecting, a confidence level, and a statistical power level. The baseline comes from your recent sends. The improvement is a business decision. For confidence and power, the common defaults are 95 percent confidence and 80 percent power, and they work fine for outbound.

Your baseline should come from real data. Pull the reply rate for the same audience and sequence step over the last few months from your sequencing tool or your CRM. If you do not have that, run your current email to a few hundred contacts first, just to measure the baseline.

The minimum improvement is the input people skip, and it matters most. Ask yourself what lift would actually change a decision. If moving from 3 percent to 3.5 percent would not change anything about your motion, do not design a test to detect it. Picking a bigger, meaningful lift keeps your sample size realistic.

How do you set up the calculation in Google Sheets?

Put your four inputs in four cells, then use one formula that applies the standard two-proportion sample size calculation. Google Sheets lists NORM.S.INV in its function list, and it turns your confidence and power levels into the numbers the formula needs. The result is the number of contacts each version must reach.

Start with a new sheet. In cell A1, type "Baseline reply rate" and put 0.03 in B1. In A2, type "Target reply rate" and put 0.05 in B2. In A3, type "Confidence" and put 0.95 in B3. In A4, type "Power" and put 0.8 in B4. Format B1 and B2 as percentages if you like.

In A6, type "Contacts per version." In B6, enter this formula: =ROUNDUP((NORM.S.INV(1-(1-B3)/2)*SQRT(2*((B1+B2)/2)*(1-(B1+B2)/2))+NORM.S.INV(B4)*SQRT(B1*(1-B1)+B2*(1-B2)))^2/(B1-B2)^2,0). Press enter. With these inputs, the sheet returns 1,506.

What do the results look like for common reply rates?

With 95 percent confidence and 80 percent power, detecting a jump from 3 percent to 5 percent needs about 1,506 contacts per version. Going from 3 percent to 4 percent needs about 5,301. Going from 3 percent to 6 percent needs about 749. From 5 percent to 8 percent needs about 1,059. Small lifts are expensive to prove.

Those numbers come straight from the formula above, so you can check them in your own sheet. The pattern is the important part. Halving the lift you want to detect roughly quadruples the volume you need. That is why testing tiny wording tweaks in cold email rarely produces a trustworthy answer for a small team.

Look at your monthly send volume next to these numbers. If your whole team sends 2,000 new contacts a month to one segment, a test that needs 5,000 per version would take five months. By then, your list, your offer, and the market have all changed. That test is not worth running in that form.

What should you test if your volume is too small?

Test bigger changes. Instead of two subject lines that differ by one word, test two different angles, two different offers, or two different segments. Bigger changes tend to produce bigger differences in results, and bigger differences need much smaller samples to detect. Low volume is a reason to test boldly, not to stop testing.

For example, compare a problem-led email against a proof-led email with a short customer result. Or compare targeting heads of RevOps against targeting founders at the same companies. These choices can move reply rates by several points, which puts them within reach of a small team's volume.

You can also change what you measure. Positive replies and booked meetings matter more than total replies, but they are rarer, so they need even more volume. Some teams track a leading signal, such as reply rate, to choose a direction, then confirm with meetings over a longer period. My piece on measuring reply quality instead of reply rate covers that balance.

How do you run the test without breaking it?

Split contacts randomly between versions, keep everything else identical, and do not stop until both versions reach the sample size you calculated. Random assignment matters most. If one version goes to Monday sends and the other to Friday sends, you are testing the day of the week, not your email.

In practice, I assign the version at the list level before anything is loaded into the sequencing tool. In Clay or a Google Sheet, add a column with a random number, sort by it, and label the top half A and the bottom half B. That way the split is random and you have a record of who got what.

Keep sender, send time, domain, and follow-up steps the same for both versions. Write down the start date, the sample size, and the metric you will judge on before the first email goes out. If you change any of those midway, the test resets. Written rules make it much harder to bend them later.

How do you read the result once the test finishes?

Compare the reply rates only after both versions reach the planned sample size, then check whether the difference is at least as large as the minimum lift you designed for. If it is, you have a reasonable winner. If it is smaller, treat the result as a tie and keep the version that is easier to run.

A tie is a useful result. It tells you the change you tested does not matter much for this audience, so you can stop spending energy on it. I would much rather have an honest tie than a fake winner that sends the team chasing the wrong idea for a quarter.

Record every test in one place with its inputs, sample size, result, and decision. A simple log in the same Google Sheet works. If you already track outbound in a sheet, my guide to setting up a reply rate dashboard in Google Sheets shows how to keep results and tests side by side.

What should you do next?

Build the four-input sheet today and run your next planned test through it before launch. If the required sample size is bigger than you can send in a month, test a bolder change instead. Then randomize the split, fix your rules in writing, and wait until both versions reach the number.

Once your team gets used to this, test quality goes up fast. Fewer tests, each one big enough to trust, will teach you more than a dozen tiny tests that never had a chance. If you want a way to score the emails themselves, a simple team scorecard pairs well with this kind of testing.

I build websites and the automations behind them for lead generation, and outbound testing systems are part of that work. If you want help setting up tests your team can actually trust, reach out and let's chat.