An Introduction to Benchmarking (UX)

The Process of Benchmarking UX Research

Each benchmark study should follow a structured plan to collect data at set intervals over time. By running the same tasks on upgraded products or gathering fresh data from the same sources assures continuity.

To start benchmarking, we need to consider the following:

1. What to measure – defining the key UX metrics.
2. How to measure them – tools, tasks, and research methods.
3. Collecting data to establish a baseline and benchmark.
4. Redesign the product or action changes to your operation.
5. Collect new raw data and interpret the results.
6. Where possible, calculate ROI.

A benchmark study is rarely a one-time effort, as these studies work best as part of a cycle. Each iteration of testing can be considered a new baseline for further improvements, implemented through further design efforts and exploration. In our list, a cycle may repeat points 4 to 6 with each change or addition.

Identifying Key Metrics and Goals in UX Benchmarking

In UX, we utilize various research methods to gather essential metrics.

Typically, quantitative data—numbers, rates, times, and measurable scores—can be quick and easy to extract from analytics sources and measurable sources, whereas qualitative data—more often assigned to behavioral and attitudinal reactions: user satisfaction, experience, and usability—lean more towards surveys, interviews, and the other UX research tools we use to gather user insights.

User satisfaction depends on multiple areas of operation and a range of UX metrics:

• Do they like your new or upgraded product? (customer satisfaction)
  Measure satisfaction and ease-of-use ratings, and NET promotor score.
• How involved are users with the product/changes? (user engagement)
  Measure time on task, usage, and conversion rate.
• Do they explore the changes or features? (adoption)
  Measure sales figures, conversions, visitor numbers, or new accounts.
• Do users stay faithful to the product? (retention)
  Measure retention rates, returning users, churn, and renewals.
• Are updates or product introductions efficient and effective?
  Measure success rate, completion rate, error count, and time on task.

We use a range of metrics to determine the ROI of upgrades and new features, but there are times when specific metrics show the value of those changes to the user against previous performance. Measuring success might mean fewer customer service requests or process abandonments, lowering the average time on task for some events, or improving the task time where engagement is the key metric.

Ultimately, these areas add up to high levels of customer satisfaction, which feed through to the bottom line over time.

Selecting Tools and Methods for UX Benchmarking Research

Each benchmarking study is unique, and the UX metrics that dictate their baseline and achievements reflect that.

For product studies, UX benchmarking involves many of the same tools as usability testing, with plenty of data sources we can turn to for quick and easy information:

• Web and CRM analytics provide essential data surrounding conversions, sales, leads, time-on-task-based operations, and more.
• Heatmaps show UX designers how and where users engage with content.
• Usability testing gathers specific task data and user insights with an option to gather qualitative data through additional questionnaires for new and existing users.
• User research—predominantly questionnaires and surveys—focuses on UX metrics such as customer satisfaction score, usability, and net promotor score.

For competitive analysis, UX benchmarking utilizes websites and tools that compare websites, apps, and products against one another, or we can use prototype replicas to test a competitor’s product against a client’s.

Similarly, exploring industry standards for benchmark data means engaging with research agencies and industry reports to uncover the figures that determine how you rank in your field and where you need to change to improve.

Stakeholder-imposed benchmarks are most often set according to business goals. It’s not always solely about improving the bottom line or boosting sales—although that’s usually the ultimate goal—but improving stages in the process. In these situations, benchmark studies may focus on reducing technical support, time-to-market, errors and abandonment, or improving customer loyalty, satisfaction, and retention.

Best Practices in Gathering and Analyzing Data

Combining quantitative usability testing with qualitative data – A persuasive set of figures might look like compelling evidence, but only if there’s a why behind the what. Taking the easy route of solely exploring analytics figures (typically quantitative data) can help to point an accusatory finger, but only when we engage in each service’s user experience (qualitative data) do we find the reasons behind those figures.

Expertise is invaluable – Employing professional UX benchmarking involves catering to all angles. Attitudinal and behavioral data show where customer experience is failing, uncovering the areas that need improvement.

Presenting the results – The other area where UX professionals exceed expectations is presenting the results with a final report that tells the whole story. Presenting a clear and accurate comparison between the past performance of an earlier version and an upgraded user experience greatly benefits stakeholders and UX designers.

Whether you discuss all the variables in your report or not, including them in your report’s appendix is essential. It’s another way to show your holistic approach and understanding of the wider issues and environments.

UX Benchmarking Techniques

Comparative Benchmarking: Analyzing Competitors

Competitor product benchmarking is a particular area of our expertise. If you’d like to know more, follow the link to our services page.

When we look at competitor performance, we consider the following:

• What do they do better? And how do they do it?
• What data do they focus on, and how does that affect their results?
• What changes must be made to match their performance and results?
• How much of the relevant data is available?
• Are there specific patterns that connect to their results?

Performance Benchmarking: Evaluating User Experience

Performance benchmarking studies typically include existing users and how we can make performing the same task a better experience.

• Are technical issues causing problems?
• Have recent changes affected lower engagement or conversion figures?
• Are these benchmarking studies designed to solve underperformance or structure ways to enhance high-performing areas?

Historical Benchmarking: Tracking UX Changes Over Time

We use every benchmark study to track changes over time. As with many UX studies, they create a cycle of research that promotes UX design improvements and is a continuing process.

• What were the key issues with earlier versions of our website, app, or product?
• How can we continue to improve user experience for existing users and our target audience?

Conclusion

UX benchmarking provides beneficial opportunities to measure progress against a meaningful standard. No hard and fast rule dictates what a study should contain or be expected to reveal, only to advance the operation of the product and enhance the user experience.

UX research takes the risk out of making unexplored changes, and UX benchmarking is no exception. With a defined starting point, it’s a highly effective method and an opportunity to conduct tests that provide new solutions for enhanced user experiences.