Experiments are powerful ways to validate your assumptions. By setting up your experiment to collect relevant KPI’s you and your business will get deeper insights that you can put into action. In this article, we will outline a step-by-step process for running a data-backed experiment.
The following outline is a paraphrase from chapter 8 in “Winning with Data” by Tomasz Tunguz and Frank Bien. Anyone looking to build a data-centric culture should take a look at this book. The authors do an excellent job of showing the power of big data and how using this data can improve your business and culture. Ok, let’s get into the steps to building a data-backed experiment.
Before ever running an experiment, you need to be sure that your outcomes will produce actionable insights. If your experiment doesn't help you make better decisions, you may need to rethink whether or not this experiment is worth doing in the first place.
After you have determined that the experiment is worth undertaking, the next step is to bookend expected results. Colin Zima, Chief Product Officer at Looker, explains, “This is one of the keys to ensure rigor around decision making.” An example from “Winning with Data” highlights Zendesk's Net Promoters Scores. If the score falls below 50, the team investigates whether the sales team is over-promising in the sales process. This is one example of how a team bookends expected results to help them take action.
When creating a statistical experiment, we need to develop a hypothesis. This gives us something to validate or invalidate. But, we also have to be careful of biases. When designing the experiment, be sure to allow room for “the devil's advocate.” This will help ensure that the results are more accurate.
The experiment should be designed to collect as much data as needed to attain statistical significance. This will differ from company to company and experiment to experiment. To come to a conclusion that is statically relevant, you need to have enough data points.
For example, when we test whether or not a landing page is converting at a high enough rate, a sample of 10 users is not nearly enough. Typically we shoot for a minimum of 100. At this threshold, we start to have enough data to conclude whether or not the page is converting at a healthy rate.
For more in-depth experiments, you may also want to find the probability of your hypotheses or P-Value. A P-Value is used in hypothesis testing to help you support or reject the null hypothesis. The P-Value is the evidence against a null hypothesis. The smaller the P-Value, the stronger the evidence that you should reject the null hypothesis. You can learn more about p-value here: http://www.statisticshowto.com/p-value/
You also may need to find the Z-score. The Z-score (aka, a standard score) indicates how many standard deviations an element is from the mean. You can calculate a Z-score from the following formula.
Z is the Z-score, X is the value of the element, μ is the population
While these formulas can seem overwhelming if you haven’t worked with statistics in a while, having a base understanding of them can help. There are some programs that help you figure out these numbers to predict outcomes better.
At this point, you should have a good understanding of what you want to solve and the sample size you will need to achieve statistical significance. Now, we need to calculate the time we will need to get the results we are looking to find. Again, different experiments will take varying amounts of time. But knowing what information you need and how long it will take to get, will help your team and business have proper expectations for the experiment.
The final step is to put the experiment into motion. Once we have collected the data, we must analyze the information and validate or invalidate our assumptions. Then we must take action on the data we uncovered.
This is a simple 5-step process to getting started with data-backed experiments. While the process is simple, it is very powerful and can help you and your business get the actionable data needed to make better decisions.
We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites.