An Introduction to Online Testing

This is a post written by CPA, an active member of CPA Hustle (no relation). While does not normally publish articles of value, CPA sent this over to me, and since I have no ideas for posts, this will do.

Remember, get your grind on at

If you guys enjoy this post, there will be more in this series on proper online testing.

Proper Testing: An Introduction to Online Testing (Part 1 of the Proper Testing Mini-Series)

Online Testing has become very popular in the last two years. No matter where you go, your favorite blog, your network of friends & associates, or even the latest course on affiliate marketing, everyone is screaming, “Test Test Test!”. While it is good to test and crucial to the success of your campaign or business, it has come to my attention that 99% of those actually putting testing into practice are doing it incorrectly. You’re probably thinking, “Testing is easy so how could someone possibly be doing it wrong?”. Well, testing is more complicated than you might think. On the contrary, once done right, you can see your increase in conversions go up by hundreds of percentage points very quickly.

Why You Should Test

Let’s say you have a campaign that you are running that is having success. You are making a good ROI right from the start so why mess with it? While a site or landing page may have success initially, it can eventually fail. It is crucial to test properly and constantly to avoid running into catastrophic errors in the future. You may have heard the quote, “Adequacy is the enemy of excellence”. In other words, sufficiency should never be ‘good enough’. You should never feel satisfied with your current success and always be looking for ways to improve something even further. Satisfaction is not excellence but laziness.

An Example

Let’s say your testing a landing page for a new CPS (Cost-Per-Sale) offer you plan to launch. You are offering a 7-day free trial with a small S&H charge. You are not sure whether you should charge $2.00, 3.50, or $5.95. You might think you should charge $5.95 because that’s more money in your pocket initially than if you were only charging $2.00. Right? Wrong.

We decide to send 1,000 clicks to each price point with a CPC (Cost-Per-Click) of $1.25. Upon completion of the test, which took about 3 days, we find the following:

$2.00: 173 leads – $346.00 in revenue
$3.25: 94 leads – $305.50 in revenue
$4.95: 52 leads – $257.40 in revenue

As you can see from the results above, the $2.00 S&H fee produced 79 more leads than the next price point, an 84% increase in opt-ins. While the initial difference is only $41.50 in a 3 day period, in a month your looking at $415.00 extra. The main point here though is since your product is a continuity product you also have to calculate that into the equation. Let’s say after the 7-day free trial your customer is billed $49.95 per month at a 67% rebill rate. In a month here are the following totals:

$2.00 – 1730 leads = 1159 (67%) @ $49.95 = $57,892.05
$3.25 – 940 leads = 630 (67%) @ 49.95 = $31,468.50
$4.95 – 520 leads = 172 (67%) @ 49.95 = $8,591.40

By using the smaller price point, you have made an additional $26,423.55 by doing a simple price point test. You also have an 84% increase in leads. You would have never known this if you hadn’t tested! In a 12 month period, that simple test will help you bring in an extra $317,000 in revenue. (assuming they stick around for 12 months)

Improper Testing Vs. Proper Testing

Online Testing (such as split testing) is crucial to max out conversion rates for your page. It is important to not speculate too much but to test instead. Let the numbers show you how the customers feel. While testing is so important, it is actually better to speculate than to conduct an invalid test. Invalid testing can produce invalid results that in turn give you a false sense of confidence in how you plan to proceed. If you hadn’t tested at all at least you’d be more cautious and realize that you might be making a big mistake. If you get back data that is invalid or false, you might run with it and make your long term results even worse. Improper testing can become very expensive and ultimately plummet your chances of successfully reaching your target market.

There are a lot of other issues besides the confidence interval or statistical insignificance. You need to be able to recognize instrumentation errors and historic effects. Validity is the most important. If you don’t have good analysis tools and you don’t have a good way to interpret your data and the insights shown, you will often conduct tests and you’ll only learn one thing when you should be learning many things. You should never learn only one thing within the bounds of your test. There will always be all kinds of lateral insights you can find besides the obvious vertical result you initially were looking for.

The concept of a successful test is to not only test the right way, but to also test the right factors that will produce the best results. A successful test is a test that has been designed properly so that the resulting data provides you with an accurate projection that answers your question. All good tests, especially single factorial tests, are based around an initial research question. If your asking the wrong questions, it doesn’t matter if you get an answer since the answer itself is irrelevant to your factual needs. The goal is not to conduct an effective test, but to conduct a useful test.

Final Thoughts

If you were responsible for conducting a price test like the one above, do you really think could you do it properly? Most of you probably said “Yes” which leads me to the following questions; How would you design this test? How would you determine the right price points that you were going to test? How would you setup or lay out the treatment pages? How would you determine if you had a valid sampling size for the test?
I have a feeling most of you couldn’t answer these questions but if you could, please post your answers in the comments section below. During the next few weeks I will be continuing this mini-series on conducting accurate and valid tests for your affiliate offers, landing pages, or web sites. By the time we finish, you should have working knowledge of how to setup proper tests and produce accurate results and deem yourself an expert.

  • Noddin my head like yeah
    Movin my hips like yeah

  • Clue List

    I wanna take you to teh movieezzz, shaday.

  • Good post. What a surprise, Barman didn’t write it.

  • ^^^^ Genius. First rate comment!!!!. +1!!

  • always on point.