{"id":4497,"date":"2016-02-19T09:23:34","date_gmt":"2016-02-19T07:23:34","guid":{"rendered":"https:\/\/usersnap.com\/?p=4497"},"modified":"2025-07-30T10:15:28","modified_gmt":"2025-07-30T08:15:28","slug":"startups-stop-ab-testing","status":"publish","type":"post","link":"https:\/\/usersnap.com\/blog\/startups-stop-ab-testing\/","title":{"rendered":"Startups, please stop talking about A\/B testing!"},"content":{"rendered":"\n

Startup people, please do me a favor and stop talking about A\/B testing<\/strong>. Please.<\/p>\n\n\n\n

You might wonder: Thomas, why are you so grumpy today?<\/em> Let me tell you. It has nothing to do with being grumpy. It\u2019s just the truth. Sorry!<\/p>\n\n\n\n

Before deciding I\u2019ve gone crazy, make sure to read the following thoughts first.
\n<\/p>\n\n\n\n\n\n\n\n

What is A\/B testing?<\/h2>\n\n\n\n

A\/B testing, also known as split testing, is comparing two different versions of a web page. The goal is to evaluate which version is performing better.<\/p>\n\n\n\n

Or as Wikipedia<\/a> states:<\/p>\n\n\n\n

\n

\u201cIn marketing and business intelligence, A\/B testing is a term for a randomized experiment with two variants, A and B, which are the control and variation in the controlled experiment.\u201d<\/p>\n<\/blockquote>\n\n\n

\n
\"ab<\/a><\/figure>\n<\/div>\n\n\n

You get there by showing two different versions of a web page to a group of people who visit your site during a specific period of time. The two versions can only differ in the color of the Call-to-action button, or in some cases, you could a\/b test two completely different design drafts.
\n<\/p>\n\n\n\n

See the difference in these two versions?<\/em><\/p>\n\n\n\n

Basically, you can run A\/B tests on almost anything on your website that affects visitors\u2019 behavior.<\/p>\n\n\n\n

The version that gains better conversion rates wins.<\/p>\n\n\n

Try Usersnap for Testing<\/h2>Try Usersnap Now<\/a><\/div>\n\n\n\n

Why running A\/B tests?<\/h2>\n\n\n\n

Running AB tests can help you make decisions about certain design drafts, copy ideas or arrangement of certain sections on your website.<\/p>\n\n\n\n

You can focus on one idea at a time, collect data about the impact of your question and make better decisions on the basis of the A\/B test. Sounds easy, huh?<\/p>\n\n\n\n

However, there are a couple of constraints which should be considered.<\/p>\n\n\n\n

All about conversion rates<\/h2>\n\n\n\n

In theory, it sounds great. A\/B tests help you make better decisions and improve conversion rates on your site. Isn\u2019t that awesome? That\u2019s what we all want, right?<\/p>\n\n\n

\n
\"stop<\/a><\/figure>\n<\/div>\n\n\n

As someone who\u2019s thinking about conversion rates on a daily basis, I see the clear benefit of thinking about ways to improve your conversion rate on your website or online shop.<\/p>\n\n\n\n

And A\/B tests might be the way to do that.<\/p>\n\n\n\n

The challenges of A\/B tests for startups<\/h2>\n\n\n\n

However, there are a couple of things that people, especially in the startup ecosystem, need to consider when running A\/B tests. Nonetheless, if you are not a specialist with a top standard of expertise in the field in which you are attempting to succeed, you should always consider hiring startup consulting<\/a> services before you go any further.<\/p>\n\n\n\n

The problem of quantitative data<\/h2>\n\n\n\n

I think the most important thing when it comes to A\/B testing is data. And the amount of data available.<\/p>\n\n\n\n

As I mentioned before, there\u2019s nothing wrong of running A\/B tests as long as you have enough data to work with when conducting A\/B tests. My guess is that as a startup that just released a new product or landing page, you probably do not have that many visitors on your site.<\/p>\n\n\n\n

My guess is that as a startup that just released a new product or landing page, you probably do not have that many visitors on your site. As both versions must be tested simultaneously<\/strong>, you need a certain number of visitors on your site in order to measure relevant results.<\/p>\n\n\n\n

An A\/B test is a proper tool for testing variants on your site if you have a couple of thousand daily visitors. It\u2019s not a proper tool if you just have a couple of daily users. As most startups can be found in the second scenario, I won\u2019t bet my money on those A\/B tests.<\/p>\n\n\n\n

The problem of significance<\/h2>\n\n\n\n

Most A\/B tests run for a limited time. It\u2019s important to define that period in advance in order to create meaningful results.<\/p>\n\n\n\n

I absolutely recommend that everyone use an A\/B test calculator (like this one<\/a>) to help determine exactly how long to run a test before giving up.<\/p>\n\n\n\n

If you are dealing with a low number of daily visitors, you\u2019ll be surprised how long your tests needs to run.<\/p>\n\n\n\n

Here\u2019s an example:<\/strong>
\nLet\u2019s say you have 250 daily visitors<\/em> on your site and you\u2019d like to run an A\/B test on all your visitors.<\/p>\n\n\n\n

You have an existing conversion rate of 5%<\/em> and you\u2019d expect a minimum improvement in conversion rate by 15%<\/em>.<\/p>\n\n\n\n

All in all, you will need 162 days<\/strong> to run your A\/B test in order to get a significant result. And 162 days can be a long period for startups.<\/p>\n\n\n

\n
\"ab<\/a><\/figure>\n<\/div>\n\n\n

Yep – that is a long time. Give this calculator<\/a> a try and see how long an A\/B test must run in your case. In this blog post<\/a>, you can learn more on statistical analysis and A\/B testing.<\/p>\n\n\n\n

The problem of your non-defined target group<\/h2>\n\n\n\n

In the early stages of your startup and\/or product, chances are high that you do not know a lot about your (potential) customers. Maybe you aren\u2019t even sure what your ideal customer profile<\/a> is or what it should be.<\/p>\n\n\n\n

You may have just released a first version of your product and gained some traction among certain niches or people. Running A\/B tests in this early stage of your product might bring misleading results. Do yourself a favor and start thinking about the bigger picture here.<\/p>\n\n\n\n

Without a proper definition of your Ideal Customer Profile (ICP)<\/em>, running an A\/B test won\u2019t get you anywhere.<\/p>\n\n\n\n

The problem of not being a scientist<\/h2>\n\n\n\n

To startups and marketers, A\/B tests may sound sexy. However, it shouldn\u2019t prevent you from thinking like a scientist.<\/p>\n\n\n\n

Running an A\/B test might be easy but a scientist will tell you that it\u2019s not about the test run itself. Just like scientists, you should care mostly about one thing: The result must be correct.<\/p>\n\n\n\n

However, getting correct results can only be achieved by putting enough time and effort into it, which many startups and marketers do not have.<\/p>\n\n\n\n

For additional reference, I recommend this article on How Not To Run An A\/B Test<\/a>.<\/p>\n\n\n

Try Usersnap for Testing<\/h2>Try Usersnap Now<\/a><\/div>\n\n\n\n

The problem of your MVP<\/h2>\n\n\n\n

Having just released your minimal viable product, chances are high that your product itself will change over time. As your product changes, your website and website visitors wil change as well.<\/p>\n\n\n\n

A\/B testing might be problematic when the results of certain tests are influenced by your changing product.<\/p>\n\n\n\n

An alternative to A\/B testing for startups<\/h2>\n\n\n\n

I can\u2019t emphasize enough how great A\/B testing can be if well executed. However, it just might not be the golden egg that some startup founders and people expect it to be.<\/p>\n\n\n\n

Instead of running A\/B tests for your startup, I recommend creating so-called experiments and collecting qualitative feedback from your customers and other stakeholders. As (most) startups have limited resources (time, money or haven\u2019t found the right startup business credit card<\/a>), a well-prepared and executed experiment funnel can be your way to growth.<\/p>\n\n\n\n

Here\u2019s how we are running experiments at Usersnap.<\/p>\n\n\n\n

(No, we didn\u2019t invent this on our own – we were inspired by this blog post on Growth Experiments at SoundCloud<\/a>. I won\u2019t go into too much detail here so please check out the blog post if you want to know more.<\/em>)<\/p>\n\n\n\n

Step 1: Brainstorming experiments<\/strong><\/h3>\n\n\n\n

In order to improve certain outputs (like conversion rate, sign-ups, new customers) you need to think about the input required to achieve that. As a first step, you need to come up with ideas on how to make the output happen.<\/p>\n\n\n\n

All ideas should go into a backlog, in our case a simple google spreadsheet.<\/p>\n\n\n\n

Step 2: Prioritize<\/strong><\/h3>\n\n\n\n

Having collected a few ideas, you should start prioritizing. Take a look at your list of experiment ideas and prioritize by considering the following three things:<\/p>\n\n\n\n