The A/B Testing Culture and Process

In the previous post of this series, you got introduced to what A/B testing is. Before we deep dive into the A/B testing, It is important to understand few factors to be considered for successful experiments.

In this post, you will understand about the culture needed to encourage testing and an intro to the process of A/B testing. The key to run an A/B test is to foster the culture of experimenting and to discipline a process to test.

Fostering the Culture

Experimenting and executing tests needs a culture that encourages it. In a talk (video) by Hazjier, Global Head of Strategy at Optimizely, he measures this culture on a scale of “Freedom”. This refers to the level flexibility and liberty teams have to experiment. The following image explains 3 different levels of freedom.

Scale of Freedom to Test

While the levels of freedom cannot be compared because it depends on the company policies and processes, it influences the number and type of tests that can be tried.

He quotes Booking.com , a travel company we are familiar with, as an example for the high level of freedom. It tested the brand names “booking” and “booking.com” to decide the better converting name!

booking

It should test and solve your real problems. A nice example is highlighted in the talk quoted above. A shirt brand tested a male model with different level of beards to be featured on the website.

beard

The Beard 6 variant drove more than twice the conversions by the Beard 1 variant. However, this result cannot be applied across the website as it might not become interesting and users may not click on them. Hence, the problems or hypothesis you test must be prioritized based on the impact of its result.

The Process

The A/B testing has to be run as a scientific process. The results needs to be statistically concluded. The statistical significance makes sure the results are fair covering all scenarios like null hypothesis and the factors affecting users browsing behavior.

The A/B testing process involves the following 6 steps. This is a common framework that can be adapted and modified with additional steps if needed.

A/B Testing Process Iteration

Every step in the process requires certain tools, team work and thought process. Each of these steps will be detailed out in the forthcoming blogs of this series.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s