Tuesday, 25 February 2014

Doing A/B Testing? Read This

This article claims that "MOST WINNING A/B TEST RESULTS ARE ILLUSORY”. I agree.

Please ignore the well-meaning advice that is often given on the internet about A/B testing and sample size.
For instance, a recent article recommended stopping a test after only 500 conversions. I’ve even seen tests
run on only 150 people or after only 100 conversions [5]. This will not work. The truth is that nearer 6000
conversion events (not necessarily purchase events) are needed.

and

Almost two-thirds of winning tests will be completely bogus. Don’t be surprised if revenues stay flat or even go down after
implementing a few tests like these.

Read it all here (PDF).

No comments:

Post a Comment

Note: only a member of this blog may post a comment.