Doing A/B Testing? Read This

This article claims that "MOST WINNING A/B TEST RESULTS ARE ILLUSORY”. I agree.

Please ignore the well-meaning advice that is often given on the internet about A/B testing and sample size.
For instance, a recent article recommended stopping a test after only 500 conversions. I’ve even seen tests
run on only 150 people or after only 100 conversions [5]. This will not work. The truth is that nearer 6000
conversion events (not necessarily purchase events) are needed.

and

Almost two-thirds of winning tests will be completely bogus. Don’t be surprised if revenues stay flat or even go down after
implementing a few tests like these.

Read it all here (PDF).

Comments

Popular posts from this blog

WordPress comments can never be really disabled

Ask Customers for Money Already...

What is Dubbiya? Rethinking Entity-Relationship Diagrams