In UX terms, benchmarking is about monitoring progress or change. It’s achieved by comparing new qualitative and quantitive data to previous results, industry standards, or competitor performance.
These benchmarks are the baseline metrics by which we judge perceived progress and success.
For example, UX24/7 was commissioned to conduct a multi-platform UX benchmarking study for a global gaming company operating in three different international markets. As a global leader in their industry, we were asked to explore five products in three countries, pinpointing customer frustrations and improving the experience. We utilized our experience in benchmarking, localization, and user journeys to reveal problem areas, including poorly localized payment pages that created immediate barriers and abandonment, outdated visuals, and legacy system issues.
We’ve carried out benchmark studies for O2, reviewing competitors’ apps, and also for eBay, where we reviewed one of their competitors over multiple markets.
Understanding UX Benchmarking: Definition and Importance
UX benchmarking is a comparative evaluation measured against a pre-determined standard. By identifying a baseline or benchmark, we can determine performance through the analysis of measurable data.
We use the UX benchmarking process to set and achieve predetermined goals—issued by departments or stakeholders—to improve product performance, user satisfaction, and ROI.
When conducting a UX benchmarking study, an essential part of the process is to decide precisely what you need to measure, the research method to gather specific metrics, and how to determine the quantifiable difference.
The Role of Benchmarking in UX Research and Design
Typically, we perform UX benchmarking studies to track relative performance in the three following areas:
- The progress of a product, service, operation, or business.
- To directly compare to competitors, industry standards, or a stakeholder-determined goal.
- To show the value of UX practices, design efforts, and projects.
At the highest end of the scale, benchmarking evaluates how well our businesses perform; looking deeper into products and services, it can show how a single design change, tool, or element improves (or fails).
The Process of Benchmarking UX Research
Each benchmark study should follow a structured plan to collect data at set intervals over time. By running the same tasks on upgraded products or gathering fresh data from the same sources assures continuity.
To start benchmarking, we need to consider the following:
- What to measure – defining the key UX metrics.
- How to measure them – tools, tasks, and research methods.
- Collecting data to establish a baseline and benchmark.
- Redesign the product or action changes to your operation.
- Collect new raw data and interpret the results.
- Where possible, calculate ROI.
A benchmark study is rarely a one-time effort, as these studies work best as part of a cycle. Each iteration of testing can be considered a new baseline for further improvements, implemented through further design efforts and exploration. In our list, a cycle may repeat points 4 to 6 with each change or addition.
Identifying Key Metrics and Goals in UX Benchmarking
In UX, we utilize various research methods to gather essential metrics.
Typically, quantitative data—numbers, rates, times, and measurable scores—can be quick and easy to extract from analytics sources and measurable sources, whereas qualitative data—more often assigned to behavioral and attitudinal reactions: user satisfaction, experience, and usability—lean more towards surveys, interviews, and the other UX research tools we use to gather user insights.
User satisfaction depends on multiple areas of operation and a range of UX metrics:
- Do they like your new or upgraded product? (customer satisfaction)
Measure satisfaction and ease-of-use ratings, and NET promotor score.
- How involved are users with the product/changes? (user engagement)
Measure time on task, usage, and conversion rate.
- Do they explore the changes or features? (adoption)
Measure sales figures, conversions, visitor numbers, or new accounts.
- Do users stay faithful to the product? (retention)
Measure retention rates, returning users, churn, and renewals.
- Are updates or product introductions efficient and effective?
Measure success rate, completion rate, error count, and time on task.
We use a range of metrics to determine the ROI of upgrades and new features, but there are times when specific metrics show the value of those changes to the user against previous performance. Measuring success might mean fewer customer service requests or process abandonments, lowering the average time on task for some events, or improving the task time where engagement is the key metric.
Ultimately, these areas add up to high levels of customer satisfaction, which feed through to the bottom line over time.
Selecting Tools and Methods for UX Benchmarking Research
Each benchmarking study is unique, and the UX metrics that dictate their baseline and achievements reflect that.
For product studies, UX benchmarking involves many of the same tools as usability testing, with plenty of data sources we can turn to for quick and easy information:
- Web and CRM analytics provide essential data surrounding conversions, sales, leads, time-on-task-based operations, and more.
- Heatmaps show UX designers how and where users engage with content.
- Usability testing gathers specific task data and user insights with an option to gather qualitative data through additional questionnaires for new and existing users.
- User research—predominantly questionnaires and surveys—focuses on UX metrics such as customer satisfaction score, usability, and net promotor score.
For competitive analysis, UX benchmarking utilizes websites and tools that compare websites, apps, and products against one another, or we can use prototype replicas to test a competitor’s product against a client’s.
Similarly, exploring industry standards for benchmark data means engaging with research agencies and industry reports to uncover the figures that determine how you rank in your field and where you need to change to improve.
Stakeholder-imposed benchmarks are most often set according to business goals. It’s not always solely about improving the bottom line or boosting sales—although that’s usually the ultimate goal—but improving stages in the process. In these situations, benchmark studies may focus on reducing technical support, time-to-market, errors and abandonment, or improving customer loyalty, satisfaction, and retention.
Best Practices in Gathering and Analyzing Data
Combining quantitative usability testing with qualitative data – A persuasive set of figures might look like compelling evidence, but only if there’s a why behind the what. Taking the easy route of solely exploring analytics figures (typically quantitative data) can help to point an accusatory finger, but only when we engage in each service’s user experience (qualitative data) do we find the reasons behind those figures.
Expertise is invaluable – Employing professional UX benchmarking involves catering to all angles. Attitudinal and behavioral data show where customer experience is failing, uncovering the areas that need improvement.
Presenting the results – The other area where UX professionals exceed expectations is presenting the results with a final report that tells the whole story. Presenting a clear and accurate comparison between the past performance of an earlier version and an upgraded user experience greatly benefits stakeholders and UX designers.
Whether you discuss all the variables in your report or not, including them in your report’s appendix is essential. It’s another way to show your holistic approach and understanding of the wider issues and environments.
UX Benchmarking Techniques
Comparative Benchmarking: Analyzing Competitors
When we look at competitor performance, we consider the following:
- What do they do better? And how do they do it?
- What data do they focus on, and how does that affect their results?
- What changes must be made to match their performance and results?
- How much of the relevant data is available?
- Are there specific patterns that connect to their results?
Performance Benchmarking: Evaluating User Experience
Performance benchmarking studies typically include existing users and how we can make performing the same task a better experience.
- Are technical issues causing problems?
- Have recent changes affected lower engagement or conversion figures?
- Are these benchmarking studies designed to solve underperformance or structure ways to enhance high-performing areas?
Historical Benchmarking: Tracking UX Changes Over Time
We use every benchmark study to track changes over time. As with many UX studies, they create a cycle of research that promotes UX design improvements and is a continuing process.
- What were the key issues with earlier versions of our website, app, or product?
- How can we continue to improve user experience for existing users and our target audience?
UX benchmarking provides beneficial opportunities to measure progress against a meaningful standard. No hard and fast rule dictates what a study should contain or be expected to reveal, only to advance the operation of the product and enhance the user experience.
UX research takes the risk out of making unexplored changes, and UX benchmarking is no exception. With a defined starting point, it’s a highly effective method and an opportunity to conduct tests that provide new solutions for enhanced user experiences.