Doing A/B Testing? Read This

This article claims that "MOST WINNING A/B TEST RESULTS ARE ILLUSORY”. I agree.

Please ignore the well-meaning advice that is often given on the internet about A/B testing and sample size.
For instance, a recent article recommended stopping a test after only 500 conversions. I’ve even seen tests
run on only 150 people or after only 100 conversions [5]. This will not work. The truth is that nearer 6000
conversion events (not necessarily purchase events) are needed.

and

Almost two-thirds of winning tests will be completely bogus. Don’t be surprised if revenues stay flat or even go down after
implementing a few tests like these.

Read it all here (PDF).

Comments

Popular posts from this blog

WordPress comments can never be really disabled

Ask Customers for Money Already...

T-10: But Retina is Different...