In the previous post of this series, you got introduced to what A/B testing is. Before we deep dive into the A/B testing, It is important to understand few factors to be considered for successful experiments.

In this post, you will understand about the culture needed to encourage testing and an intro to the process of A/B testing. The key to run an A/B test is to foster the culture of experimenting and to discipline a process to test.

Fostering the Culture

Experimenting and executing tests needs a culture that encourages it. In a talk (video) by Hazjier, Global Head of Strategy at Optimizely, he measures this culture on a scale of “Freedom”. This refers to the level flexibility and liberty teams have to experiment. The following image explains 3 different levels of freedom.

While the levels of freedom cannot be compared because it depends on the company policies and processes, it influences the number and type of tests that can be tried.

He quotes Booking.com , a travel company we are familiar with, as an example for the high level of freedom. It tested the brand names “booking” and “booking.com” to decide the better converting name!

It should test and solve your real problems. A nice example is highlighted in the talk quoted above. A shirt brand tested a male model with different level of beards to be featured on the website.

The Beard 6 variant drove more than twice the conversions by the Beard 1 variant. However, this result cannot be applied across the website as it might not become interesting and users may not click on them. Hence, the problems or hypothesis you test must be prioritized based on the impact of its result.

The Process

The A/B testing has to be run as a scientific process. The results needs to be statistically concluded. The statistical significance makes sure the results are fair covering all scenarios like null hypothesis and the factors affecting users browsing behavior.

The A/B testing process involves the following 6 steps. This is a common framework that can be adapted and modified with additional steps if needed.

Every step in the process requires certain tools, team work and thought process. Each of these steps will be detailed out in the forthcoming blogs of this series.

What is A/B Testing?

A/B testing is comparing two versions of a web page, button, hero image, newsletter or any component and compare which performs better. The decision is made using goals to compare against each other. The variation with better goal conversion wins.

Multivariate testing is the process of testing more than 2 variations of the component. This blog by Kissmetrics is an interesting post on A/B and Multivariate test – blog.

Conversion Metrics

The conversion can differ between websites based on the purpose of the website. Examples:

• Sign-Ups to newsletter for a publisher
• Purchase for e-commerce
• Outbound click to the target site for an affiliate
• Time on site for publisher
• Form submission for lead generators

Components to test

While the goals is very important to decide the winner, the component that will be tested is also equally important.

Given the capabilities of tools in market, components like the following can be easily tested:

• Hero Images on home page or landing pages
• Call to action buttons
• Form elements (form length)
• Menu options
• Text in the titles, paragraph headings, the paragraph content etc.,
• Deals shown to users
• Newsletter email subject and content
• The price tiers and messaging for subscription or a product

This post is kept to be a simple intro to what an A/B testing is. This will be followed by posts on the A/B testing process, Implementing the test, Stats and factors involved in deciding the winner and few case studies from prominent A/B testing tools in the market.