Skip to content

First steps with A / B tests (to generate sales)

--Advertisements --

A / B testing can improve conversions on an ecommerce site. There are many test platforms – do a search on Google for "A / B Testing".

To run a test, identify a key conversion element, such as a product description. Half of your traffic would see version A (which is usually the control group that is unchanged) and half would see version B. The version with the highest conversion rate would be the winning element to adopt for any the traffic

A / B tests are easy to perform, typically. You can see results as fast as a few days, or even hours, depending on your traffic. There are some challenges, though. These include knowing what to test and decide when the data is conclusive.

What to test?

E-commerce companies, in my experience, can test button colors, fonts, product descriptions, images, promotions and many other items. Never test two elements at the same time if they are not combined. For example, if you are testing button colors, do not test a new product description either because you will not know which results have been generated. However, you can test the color of the button and a different label on the button.

Develop a global matrix for your tests. Here is an example.

  • Brainstorm all the potential elements to test.
  • Determine the type of test, such as design test, messaging test, promotion test.
  • Identify the pages. Determine where to run the test, such as the home page, product page, category, or the Contact Us page.
  • Create landmarks. Pull the average daily traffic and the average conversion rate before the test, for the reference points.
  • Prioritize testing. Using the benchmarks, consider running different tests on different pages, assuming the traffic is sufficient.
See also  Enable HTTP / 2 for happier customers, better SEO


Elements to Test Type of Test Pages Traffic Marking Conversion of the reference index Evaluate
Color of the Buttons Design Product page 5,000 1.20%
Characteristics of the product A Product Main page 20 000 0.85%
Product B Characteristic Product Main page 20,000 0.85%
New Category Series Main Page 20,000 0.85%
Other fonts Design Anywhere 20,000 0.85%
Award Ceremony Promotion Category Page 15,000 1.00%
Return Policy of the Banner Conception Product page 5,000 1.05%

-- Advertisements --

Provisional data

Once you have developed a matrix and started the test, make sure you have enough data to make a decision. It can be difficult. The first step is to identify the amount of traffic needed. To do this, multiply your conversion rate by the number of sessions you receive a day, a week or a month. This will provide the typical number of sales. This number should be significant enough to apply tests A and B.

A winning test that produces a sales increase of at least 10% is usually enough to go from control to testing. But for some tests, a 5% increase may be enough, especially for high volume companies.

Here are some examples.

Test 1: A small amount of traffic with a big difference in results. One week of tests with 2000 sessions produces these results:

  • Control Group A: Conversion rate of 1.5%.
  • Test group B: Conversion rate of 1.75%.
  • Difference in conversion rate: 16.7% representing five sales.
  • Conclusion: Enough data to decide to go to test group B

Test 2: Very popular site with a small difference in results. One day of tests with 50,000 sessions produces the following results:

  • Comparison group A: conversion rate of 1.25%.
  • Test Group B: Conversion rate of 1.30%.
  • Difference in conversion rate: 4% representing 50 sales.
  • Conclusion: While the percentage change is small, because of the high traffic, it is worth making the change.

Test 3: Only 1,000 sessions in the test, with the following results:

-- Advertisements --
  • Control Group A: Conversion rate of 1.5%.
  • Test Group B: Conversion rate of 1.7%.
  • Difference in conversion rate: 13.3%, or two sales.
  • Conclusion: Although the difference is significant, it is risky to recover because of the lack of traffic. Continue running the test or interrupt it.

There is a calculation to detect the required sample sizes. This is what is called "statistical power" – the probability that a test will detect an effect if there is an effect. The test platforms use the calculation to inform the sample sizes. For example, see "A / B Test Sample Size Calculator" of Optimizely. However, a little intuition and analysis is enough for most companies, according to my experience.

Large e-commerce stores can test several items daily. Small, low-traffic retailers can still benefit by testing on different pages, testing for longer periods, and prioritizing what to test.

The review of bids by competitors can help determine priorities. For example, if five out of seven competitors display a "free shipping" banner, consider testing it on your site.

Drive Sales

In summary, tests can improve conversion rates. Most of the leading e-commerce marketers are continually testing and modifying to drive sales. The job consists of choosing the elements, performing the tests and analyzing the results.