Having the right tools for the job just makes sense. Regarding UX research, user testing is how we gather much of our data, and the methods used to carry it out seem almost endless.
Today, we’re looking at unmoderated usability testing, how it differs from its counterpart—moderated usability testing—and the advantages and disadvantages of the system.
Moderated vs. unmoderated usability testing – what’s the difference?
The clue is undoubtedly in the question. It doesn’t take a rocket scientist to establish that a moderated test is one where the test subject is guided by a moderator (and there it is!). Whether they’re in the room or carrying out the study remotely over a video or audio call, they bring a presence to the process that changes how the participant acts and reacts.
A moderator is free to guide the user when they get off track, can answer questions, change the direction of the study, or delve deeper into specific actions and queries as they appear.
With unmoderated usability testing, however, there’s no such control. Without a moderator, the user is left entirely to their own devices to interpret the instructions delivered by the software tool in charge of the event.
Pros and cons of moderated vs unmoderated usability testing
|User focus and concentration||Possibly low||High|
|Without a moderator, test subjects are free to come and go as they please, to hurry through the study if not totally invested, or simply to complete the task and get paid for their service.|
|Speed and efficiency||Fast||Time-consuming|
|Without having to arrange and schedule physical or remote interviews and the time it takes to perform each user study, unmoderated testing is far faster and cheaper.
With pre-existing participant pools ready to go, an unmoderated test could be delivered almost immediately once the test has been constructed. A moderated test, however, can take months to organise and execute.
|Data consistency||Consistent||Open to manipulation|
|There’s little opportunity to influence actions or judgements when leaving unmoderated participants to their own devices unless the questions and tasks are written with an unintentional bias. Moderated tests have more opportunities for deviation and bias (conscious or subconscious).|
|Test scaling||Simple and extensive||Time consuming and costly|
|Where no human interaction is involved, it’s far simpler to scale unmoderated tests to as large a test group as you’d like. Automation allows you to cast your net to as vast a pool as you need in the same timeframe.
Moderated tests are restricted by time and resources resulting in smaller study groups, limited by the cost and availability of suitable moderators.
|International and remote access||Straightforward||Problematic|
|If you’re reaching out to an international audience you need to consider time zones, language barriers, cultures, technology, and more.
Remote unmoderated usability testing caters far better to each of these issues than its moderated counterpart ever could.
|Holding a conversation with your test participants lends itself perfectly to subsequent lines of questioning during the test. Of course, it’s possible to set up follow-up questions in an unmoderated test, but you’d have to be a mind reader to create the depth a moderator can administer on the fly.|
|Much of the participant quality relies on the software or service delivering an unmoderated test, so it’s advantageous that they typically have pools of possible users ready and waiting. However, unless your product is aimed at a broad market, you must rely on their participant filters.
Few will provide anything niche so any feedback could be irrelevant to those studies. Of course, if you can compile your own study pool within the study software, you’ll attain more relevant data, but that could add time and further resources to a practice aimed at saving on both.
|Depending on the aim of the study, you may be guided toward a particular type of data over another. For example, understanding behaviour often links better to qualitative data, including video and audio responses and observations. Automated transcripts can speed matters up for more extensive studies, but ideally, a moderator can experience significant findings and mark them as they go.
Survey-type studies collecting mainly quantitative data are far easier to process, especially in those extensive studies. Metrics are easy to rank or group, with times, rates, percentages, and scales delivering easy-to-transpose representation.
Which projects are unmoderated usability testability most suited to?
Sometimes having a phone call is far more efficient than sending an email or a text. Yet, how often have you sat in on a call or a meeting for an hour or so, wondering why they couldn’t send a simple email covering the same points?
It’s the same with moderated and unmoderated usability testing. Moderated tests work like conversations, where a two-way communication stream is part of the process. As a result, it lends itself to complex studies where user behaviour is difficult to understand and monitor. You can watch for body language, suspicious pauses, confusion, and more.
Unmoderated testing is suited to validating new features and concepts or where you have specific questions. In many ways, unmoderated usability testing is a deeper-diving, more tech-savvy survey. But don’t let that limit you; plenty of features are available over the full range of services.
How does unmoderated usability testing work?
You can break the process into six steps:
- Define your goals
- Choose the best software for your study
- Create the study tasks and questions
- Carry out pilot testing
- Consider how to recruit appropriate participants
- Synthesise the newly acquired data
1. Define your goals
As with every study, you must start with what you need to achieve. Without goals, you’re testing for testing’s sake.
With your goals clearly defined, your next job is to decide on the best-suited tools to gather the data. Do you need to record operations using video capture, monitoring detailed interactions and user behaviours? Or are simple questions with yes and no answers or rating scales enough? This dictates the type of user and service you’ll need.
2. Choose the best software for your study
Your goals help you decide the best plan to uncover problems and solutions and which type of software is likely the best for each task. However, not all unmoderated usability testing tools are the same, so exploring which will provide you with your ideal data is crucial.
The good news is that there’s plenty to choose from if you go down the unmoderated usability test route. Varying in functionality and price, you should be able to find one that suits your purpose. Testing the tools before committing to them for your study is vital.
3. Create the study tasks and questions
Creating study questions and task instructions is an art. With a moderated test, the moderator can step in at any point to put a participant back on track and is also available for any questions where the participant has lost their way.
There’s no such luxury with an unmoderated test, so the instruction, guidance, and questioning must be foolproof. So take it from us; writing such detailed instructions without providing hints or biases to your users isn’t easy.
If your users fail to understand any point along the test or misinterpret any instruction, it can render the entire data set useless, wasting both of your time.
4. Carry out pilot testing
Pilot testing an unmoderated usability test is essential. With moderated tests, you can intervene, reassess, or alter a process as it happens. With unmoderated tests, once delivered to your test subjects, you can do nothing about the failings or omissions you failed to spot.
You should test all technology processes, compatibilities, and functionality to ensure their intended delivery and performance. Operating pilot testing over different platforms with different users allows you more chances to discover pitfalls before the test goes live.
5. Consider how to recruit appropriate participants
If you have a specified target audience, your study must reflect that; otherwise, you’re wasting your time and money.
If you can use your own participants with your chosen software, then great. If you must explore a selection from an outside pool, you’ll need to lean heavily on the filters at your disposal.
6. Synthesise the newly acquired data
How you collect your data will determine how you synthesise it to uncover the revelations required to improve your website, app, software, or features. As with all studies, results need to be quantified and presented in easy-to-understand mechanisms and proposals.
Making sure you get the most from any testing means considering which methods will deliver the results you need. There are valuable reasons why we utilise several studies in conjunction; each tool has its strengths, as we’ve indicated with this unmoderated vs moderated usability testing article.
We’re always ready to talk if you’d like guidance into the type of studies and services best suited to your project. Just pick up the phone or drop us a line; we’ll be happy to help.