A/A Testing – Why You Need It


If you ever listened to ANYTHING that I’ve said in the past, odds are you’ve heard me tell you (or yell at you) to split test everything. 

It’s one of my core beliefs, and I hope it’s one of yours too. As a business owner, you must constantly be split testing various parts of your funnel in order to keep growing. 

And if your business isn’t growing then it’s dying as far as I’m concerned. 

So as the responsible direct response entrepreneur I know you are, there’s no doubt you’re already running A/B split tests (and if you’re not, what the heck are you waiting for?!)

But today we’re going to talk about a different type of testing–one that doesn’t get nearly as much attention but it’s just as important.

I’m talking about A/A testing. No that isn’t a typo–you read it right. A/A testing. And it’s essential to the growth of your business. 

Sure, there are some business owners who say it’s a waste of time to run A/A tests, but I’ll show you exactly why they’re wrong today. 

Because in this post, we’ll discuss what A/A testing is, why it’s so important, and how to run them so that you separate yourself from your competitors. 

What in the World is A/A Testing?

It’s a question I get asked a lot–sometimes in a shy manner from entrepreneurs who are embarrassed they don’t know about it and other times in a very angry way. Like “why the [insert any expletive you can think of] should I care about A/A testing?” 

I’ll tell you why you should care in a minute, but first let’s define it.

A/A testing is when you split an audience and then show both sides the exact same content (seriously). 

In other words, it’s like an A/B split test but you’re not testing elements like different CTAs, subject lines, leads, etc. You’re keeping everything exactly the same and simply splitting the audience. 

As you can imagine, visitors should (in theory) respond the same to each test so that there’s no clear winner.

I can already hear your response: “Well duh, thanks for sharing the obvious…why the heck are you taking time out of your day to write this??”

Show a little bit of patience, my entrepreneurial friend. I’ll tell you exactly why these tests are essential in the next section.

Why is A/A Testing So Important?

There’s one real pitfall that a lot of entrepreneurs fall into. It’s what I call the “tragedy of working with bad data” (catchy, right?)

Of course, analyzing data is what allows you to make the key decisions that drive your business forward. If you don’t have access to data, then you have no idea what’s working and what’s not–which puts you in a dangerous no-man’s land of blind guessing & checking. Not good. 

But just as problematic as working with NO data is working with BAD data. In fact, it’s even worse. Because if you have faulty data, you might think that something is working in your business when it’s actually not. So you decide to double down on one “winning” strategy only to be disappointed when it fails. All because you had access to bad data. 

And that’s the first reason why A/A testing is so essential:

Reason #1: A/A testing helps you make sure that your real split tests are actually working. 

As we mentioned earlier, the intended result of an A/A test is that both audiences react exactly the same to the identical pieces of content. 

But what if they don’t?

As an example, let’s say you split your audience into 2 separate groups and run an A/A test on your website home page to test conversions on your “book a call” CTA.

Now again, because the 2 pages in the test are exactly the same your conversion rates should also be (you guessed it) the same. 

But let’s say Group A has a 12% conversion rate and Group B only has a 4% conversion rate–and your software declares Version B the winner. 

Now you’re in a pickle, and that’s why occasional A/A testing is so important. 

Because if you run into a situation like this, then you can be fairly certain that your testing platform is configured wrong or that it’s ineffective. 

This is a critical piece of information because you now know that you need to address this situation before running actual A/B split tests. Otherwise, you’ll fall victim to the “tragedy of bad data” with unreliable results.

So when should you use A/A testing to check on the effectiveness of your actual split tests? There are basically 3 scenarios:

  1. You’re beginning to use a new A/B testing software
  2. You want to start a new implementation
  3. You notice key differences between the data from your split testing software and the data from your other analytics platform

In addition to these critical situations, It also turns out that there’s another reason why A/A testing is useful…

Reason #2: It can help you set a baseline for future A/B tests

Now, let’s say for a second that your A/A test goes according to plan and both groups have the same conversion rate for the “book a call” CTA we discussed in the last section. Good news, you don’t have to look for a new split testing software!

But there’s another benefit to running an A/A test–even when it’s “right.” Splitting the 2 groups gives you added confidence not only in your split testing software, but also in the baseline conversion rate. You can now start to run A/B tests using this initial conversion rate as your guideline. It’s the starting point that you can build on in all future tests. 

The point being, whether your A/A testing yields the same results (like they should) or  different results, you can still gain valuable information & data from both.

Now that you can see the benefits of running the occasional A/A test, you’re ready to run one yourself!

In the next section, we’ll discuss a simple step-by-step process to make sure your A/A tests are run properly. 

How to Run A/A Tests

Running an A/A test can be boiled down to 5 very simple steps, as follows:

Step 1: Split your audience into 2 groups of users.

Step 2: Make sure they’re being sent to identical web pages so the user experience of both groups is similar.

Step 3: Determine the KPI you’re testing (ex: number of visitors who click the “book a call” CTA).

Step 4: Run the test.

Step 5: Compare the results of each group. If the KPI results don’t match, then it’s time to evaluate the effectiveness of your split testing platform.

That’s all there is to it (easy enough, right?) Time for you to start implementing these tests to make sure you’re getting the cleanest possible data. 

One final quick note when running A/A tests: there’s a chance that any differences you see between the 2 groups might not indicate that your split testing platform is malfunctioning. 

Remember, there’s always an element of randomness when it comes to testing. Even if you have a statistical significance level of 95% that your test is accurate (which would be very high), there’s still a 1 in 20 chance that the results you’re seeing are due to a random occurrence. 

So just keep that in mind as a word of caution for ALL split tests you run (A/A or A/B).

But let’s make no mistake about it: if your A/A test fails you should absolutely look deeper into your split testing software to see whether there’s an issue. 

Sure, the difference might be due to a random chance, but that’s far less likely. Odds are there’s something wrong with your software that must be addressed before you run any other A/B tests.

Otherwise, you’ll be dealing with misleading data that prevents you from knowing what’s really converting (or not).


Of course, there are business owners & entrepreneurs who will strongly disagree with my perspective on A/A testing. “It’s a waste of time for entrepreneurs. We’re already stretched thin,” they’ll say. “The ROI isn’t there.”

But I think they’re wrong. Why? Because A/A testing is the best way to protect yourself against the tragedy of bad data. 

It’s no surprise that you need to constantly split test various parts of your funnel in order to win (especially as a direct response entrepreneur). 

But your A/B split tests are only as valuable as the accuracy of your data.

Bad data = unreliable results. 

And unreliable results are completely useless. 

You’ll waste hours of time, resources, and creativity trying to figure out which changes will move your business forward. 

But you’ll just stay stuck, unable to keep growing because you have no idea whether your data is useful or not.

And that’s a no-man’s land you definitely don’t want to find yourself in. Protect yourself against that disaster and set up your A/A tests ASAP. 

Join Our Newsletter


Leave A Comment