A/B Testing in UX

How to Conduct an A/B Test

Step 1: Define objectives and goals

If you fail to achieve the sign-ups, sales, leads, or click-throughs you expect, you’re probably wondering which part of your layout, promotion, copy, or navigation is causing your users to abandon the process.

You can explore those issues through testing, but you need to know why you’re running your tests. If an area of your website is underperforming and you think making a few changes will achieve the desired action to improve matters, document and test them. This is a must to keep your design team informed and understand why and what they need to test for.

• Locate problem areas – From previous testing or underperforming analytics and data streams.
• Set a clear goal – What you plan to improve.
• Determine what to test – Which elements are responsible for user pain points?
• Create a hypothesis – The assumption tied to your goal and testing process: “We believe that a stronger call to action/more prominent button position will increase enquiries/sales/conversions.”

Step 2: Create and implement variations

The essential point to remember about A/B testing is to keep things simple and not complicate your test’s two versions. We’d highly suggest that designers change only one item in each test. Why? Let’s say you’re testing a call to action button and have different images, colours, fonts, and sizes on both. However they’re rated, how would you tell which changes made the most impact? You can’t—so that’s why we tend to keep choices to a minimum.

If you need to test several elements, pitch them in separate tests. For example, in your first test, suggest two colour options; in the second, show variation in button size, and in the third, test different call-to-action phrases. Each test will deliver user preferences while removing the guesswork.

Step 3: Divide users into test groups

As with any UX research, testing should be conducted within your target audience. Split your test subjects fairly by demographic, and ensure the same sample size for each service, site, or feature version.

Step 4: Run the test and gather data

How you manage your tests will depend on what you’re testing and at which stage.

To test during the design process: You may show the same group two versions of the same component, page, navigation, etc., to see which is the most popular. This type of test requires a technique called counterbalancing. Counterbalancing ensures each creative is seen equal times in equal order, first, then second. As well as in A/B testing, it can be used to test numerous design treatments of multiple conditions.

Regarding product development, qualitative research methods explain how and why users prefer particular options. A/B testing is a great way to establish that information during the design and build process.

To test features and components of live apps and websites: The UX designers will split the traffic to a website or feature to monitor how users interact with the different versions. This is perfectly valid and provides robust measurement because of the traffic volume. However, live testing may affect your analytics data and SEO, so Google provides instructions on how UX designers can minimise such an impact.

Working with professional UX designers will guarantee that their methods ensure the performance you’ve worked hard to achieve won’t be affected by A/B or usability testing.

Step 5: Analyse and interpret results

Before you analyse anything, you must ensure you have enough data to deliver valuable insights, and that means conducting tests over a suitable period or putting your new design options in front of enough users. This is known as statistical significance/statistical analysis.

Once tests are complete, the best option from the two versions should be fairly obvious—the one most users prefer. However, deciding on the level of improvement you need to see to commit to making changes can vary. Typically, we expect (or hope for) a 90% or higher threshold. Lower thresholds aren’t as likely to produce statistically significant results and the upturn in practice your hypotheses require.

For elements that showed only marginal improvements, perhaps further testing can narrow down the issues causing your shortfall. Alternatively, if A/B testing isn’t delivering for you, perhaps it’s time to revert to other quantitative data user research methods, such as usability testing and monitoring.

Common Pitfalls and Challenges in A/B Testing

• Selection bias and skewed results
• Lack of statistical significance
• Failing to validate results with qualitative research
• Cutting corners or quitting the tests too early in the process
• Failing to spot seasonal variations

Tools for Implementing A/B Testing

There’s an abundance of A/B testing tools; many are aimed purely at professional UX designers, researchers, and others to help product managers and their teams with everyday monitoring. Google is ready to help (as always and with everything) with options to set up A/B testing in Google Analytics and Google Optimize.

If you’re considering carrying out your own A/B testing, a quick search will provide you with myriad commercial options, their key features and suitability to your venture, and those all-important user reviews.

Best Practices for Effective A/B Testing

• Base testing around goals
• Defining clear hypotheses
• Considering sample sizes and statistical significance
• Focussing on a single aspect during testing
• Practice iterative testing and continuous improvement
• Combining A/B testing with other research methods to maximise results

Conclusion

A/B testing is a fast and economical way to determine user preferences within a product’s UX. Upgrading a component or feature that has shown a negative impact in a specific area won’t just resolve a pain point but will often improve the conversion rate and ROI.

Whichever method or tool you choose, A/B testing can have a big impact on product performance. If you’d like to know more about this or any UX research and testing method, UX24/7 is ready to help.