How Long Do I Need To Run An A/B Test For?
Six weeks ago I got accepted to CXL institutes’ conversion optimisation mini-degree scholarship. It claims to be one of the most thorough conversion rate optimisation training programs in the world. The program runs online and covers 74 hours and 37h minutes of content over 12 weeks. As part of the scholarship, I have to write an essay about what I learn each week. This is my sixth report.
A/B testing for conversion rate optimization only makes sense if you have at least 1000 transactions a month. I realise this may be a bold statement for some, especially if you’re new to A/B testing, so I’m going to back that up and unpack my argument in this post.
First, a transaction doesn’t have to be a commercial transaction. It could be a sale, it can also be a signup or a download, or a phone call: it can be anything you want to optimise. If you want to optimise for signups, you need about...