This post is aimed at people who will be buying UX research and those in the decision making process. Both need to understand whether what they are being told about the number of participants to use in research is correct. Whether that is from a potential provider or from a colleague and driven by budget, quality or time considerations.
It was our mission when we launched UX247 to remove the black-magic associated with UX Research. There are many areas where in the past, consultants would draw in deep breaths over their lower teeth and offer the benefit of their individual wisdom. Thankfully, the number of participants required to deliver reliable findings in UX research is not one of those areas. So if you hear the sound of someone inhaling ready to deliver a monologue, be warned!
Google this question and you will get results from 3 to 20 and various reasons why. The basis upon which most estimates are based comes from Usability testing and there are two credible sources you should refer to as follows:
Nielsen Norman Group
Nielsen Norman Group (NN/g) literally wrote the book on usability testing. Jacob Nielsen (THE Nielsen in the name) concluded in the 90’s that 8 participants would deliver about 85% of the issues. Beyond this, the law of diminishing returns kicked in and it didn’t make sense to use more.
This number has since been revised and this article “Why you only need to test with 5 users“, published in March 2000 provided the mathematics to back it up. It is worth reading the entire article because it explains where more participants might be necessary. Crucially it also explains why you don’t need to go beyond 5, and part of the argument for this is that you should be testing frequently.
Better spend your budget on three rounds of UX Research within 5 users each that will refine your prototype each time. Than to test with 15 participants in one hit and experience reduced ROI through the diminishing returns theory.
Steve Krug – Don’t make me think
The second credible source is Steve Krug and his book, published in 2000 called “Don’t make me think“. Is it a coincidence that after this book was published with an argument for 5 users being sufficient that NN/g published the article above? Whatever you believe, this book was hugely influential if like me, you were getting into the industry around the start of the new millennium.
The reason this book was so important was that it was aimed at executives. The book was designed to be read by an executive in a 2-hour flight. Up until that point, most of the books on usability and UX were mainly academic.
The ‘BIG’ change since 2000
Back in the early 2000’s we only had to worry about one platform – Desktop (or laptop). Mobile web wasn’t a thing and tablet web wasn’t even that. So the number of participants that was being recommended was entirely based on one interface. All that changed with the arrival of the mobile web but even more so with the adoption of responsive design. Mobile web often meant that the mobile platform was being developed independently to the desktop. That meant the rule of 5 still applied as they were rarely, if ever, tested together.
Responsive design changed all that. Now it is typical that the website is being developed for all screen sizes in parallel. That has resulted in UX Research often being conducted across two days so that one can be dedicated to mobile and the other to desktop.
The lowly tablet is often ignored as it can be informed by the confluence of mobile and desktop. However, beware. I have been caught out more than once by a button appearing in the wrong place on tablet when it wasn’t tested at all. Best if you can to include a couple of users on tablet just to be safe.
So how many users now?
We still adopt the 5 rule but it is slightly nuanced. If we are running two-days of research with mobile and desktop then we expect that some of the leanings will crossover. Mobile experience will inform desktop and vice-versa.
What we say to customers is that we are recruiting 10 for 8 and will be happy with 4 “good” sessions on each platform. Overall we have more than 5 and with four good sessions on each platform we are very confident of identifying at least 80% of the issues and in most cases more. We have to keep in mind that with qualitative research something one-person does or that 10-people do can have similar significance.
When 5 isn’t enough
There are some cases where more users are required and these can broadly be described as follows:
- Complex software – where there is lots of complexity, interaction, use cases, scenarios and tasks. Think Excel vs an online store – the difference is vast.
- Disparate user groups – if the digital proposition is serving fundamentally different user groups differently, then you will need more users. For example a pharmaceutical website that is serving medical professionals at a research level and job seekers with an application process.
- Lengthy engagement – participants, and moderators, get tired so if the interaction required simply takes a lot of time then you may need to break it up into parts and test with multiple sets of participants. An example here might be a research process that leads to a purchase, then delivery, unboxing and getting started.
I hope that helps to answer the question of how many participants is the right number. If you would like to know more or have a project and are wondering how many participants you need, get in touch on +44(0)800 024624 or email us at firstname.lastname@example.org.