A/B testing, a common practice in digital marketing, may not always yield valid results due to a phenomenon known as ‘regression to the mean’. This occurs when a variable that is extreme on its first measurement tends to be closer to the average on its second measurement. This statistical concept can lead to illusory results in A/B tests, where the ‘winning’ version is often just a statistical anomaly rather than a genuinely superior option.
The paper suggests that businesses should be wary of making decisions based on a single successful A/B test. Instead, they should consider running multiple tests to confirm the results. This is particularly important when the difference between the two versions is small.
The paper also highlights the importance of understanding the underlying statistics of A/B testing. It argues that without this understanding, businesses may make incorrect decisions based on misleading results. Therefore, digital marketers need to be statistically literate to avoid falling into this trap.
Overall, while A/B testing can be a useful tool, it should be used with caution and a solid understanding of statistics to avoid making decisions based on illusory results.
Go to source article: http://www.qubitproducts.com/sites/default/files/pdf/mostwinningabtestresultsareillusory_0.pdf