Moderated usability testing

In this article, we’re considering the other side of the coin, examining the benefits of moderated (in-person) usability testing and what it means for the data you gather.

When should you use moderated usability testing?

Firstly, you can only conduct moderated testing when you have the time and resources to make it a reality. If budget isn’t a problem, we’d suggest that moderated testing delivers more specific results in almost all cases. However, life isn’t always so straightforward, and perhaps such detailed information isn’t always necessary. As UX researchers, we must pick our battles accordingly depending on the information we need to uncover.

Let’s assume the budget aligns with your needs and can cater for a suitable participant pool; what types of product testing achieve the most from this option?

Moderated usability testing is the preferred approach for complicated or complex products, often prototypes and early-stage options, and for solutions that compel users into exploratory questions of their own. It’s also ideally suited in discovery testing, examining existing services for problems and issues that may have flown under the radar.

Whether performed remotely or in person, moderated testing allows for greater observation, control, and exploration, delivering a depth of data unmatched by its unmoderated counterpart.

Moderated vs unmoderated usability testing – a quick recap

Without setting up a study lab or appropriate environment or arranging travel for participants and moderators to a suitable mutual location, unmoderated testing is quicker and less costly to run.

It’s a quick and efficient way to validate products and features using predetermined tasks or gathering data by asking specifically designed questions.

However, without a moderator to guide the study, monitoring each participant during the finer details of the operation, their manner and expressions, and pursuing deeper explanations into particular actions, designers may fail to uncover the data they hope for in more complex situations.

The benefits of moderated usability testing

Let’s consider moderated usability tests as real-time interviews built around performing particular exercises. The moderator can take ultimate control of any situation, guiding each participant to undertake the tasks that reveal how the product and user perform.

Whenever a participant is unsure, they have a primary point of contact to request and receive assistance. So, what does that mean for testing?

Higher levels of engagement

With a moderator conducting the study, each participant is far more likely to be focused on the task at hand. With unmoderated tests, participants can come and go as they please or be distracted by anyone or anything in their local environment.

With an ongoing conversation throughout testing and its tasks, each moderator can build rapport and trust, motivating participants to perform as well as they can.

It’s also an opportunity to help quieter and less confident participants have a more impactful voice.

Greater control

Live, moderated testing allows the facilitator complete control of each task and its direction. If they spot something unusual, they have an opportunity to follow a completely different path to attain unexpected or previously unseen data. They can also react to looks of confusion or frustration, asking what and why the performance wasn’t simple and how it could be made better.

Direct communication provides access to immediate follow-up questions

Not only do moderators redirect tasks when they’re heading in the wrong direction, getting matters back on track, but they also ask detailed follow-up questions based on each user’s actions and observations. This control delivers far more data into new or expected issues, allowing designers more guidance towards the most efficient and effective solutions.

Ease of physical observation, including facial expressions, mannerisms, and product use

Participants deliver tells and signs through body language, facial expressions, and mannerisms that they might not express in spoken words or written answers. Close monitoring makes pauses and problems easier to detect, something an unmoderated operation couldn’t hope to achieve.

Sometimes, the subtlest signs lead to the most valuable opportunities—items that could be easily missed in other types of testing.

The disadvantages of moderated usability testing

While the advantages are reasonably straightforward, navigating the disadvantages of moderated testing isn’t to be overlooked, especially if we’re going to get the best data from our study subjects.

A possibility to bias responses

With a moderator conducting each test and task, there’s a chance they’ll subconsciously guide participants towards the result they expect or hope for. Even using a hint of suggestion in their questioning could lead to users giving an answer they think the moderator wants to hear instead of how they actually think and feel.

Participants may act less naturally under observation

A good moderator will encourage their participants to relax and act as they would if they weren’t there. However, that’s a skill that takes practice. Even the best moderators will work with participants who get a little nervous or react less favourably to being under such close observation and supervision.

More costly and requires more resources to set up

As mentioned, costs are higher with moderated testing; as well as the financial implications, there’s the added time and effort it takes to organise and carry out each study. Given those additional time and money restraints, sample sizes tend to be smaller, which can lead to less accurate statistical data.

Delivering a moderated usability testing plan

There are many similarities when designing moderated and unmoderated usability test plans. For example, the following stages look reasonably similar to the process laid out on our unmoderated usability test page, yet the key differences tend to be around the design and delivery of the test and its tasks.

Define your goals.
Define the participant criteria and build your user pool.
Design and create your testing process.
Run pilot tests.
Conduct the finalised tests.
Synthesise the new data.
Analyse the results and present your findings.

Conducting tests in person allows for far freer scripts and guides. However, when designing your tests—and many UX research options lend themselves to every type and stage of usability testing—you’ll have to determine the most appropriate and likely to deliver the data you need.

Conducting moderated usability tests

A simple overview may look like the following:

Introduction – Run through the procedure and what’s expected, and try to relax the user.
Pre-session interview – Enquire about the participant’s history, behaviours, and habits, and ask any necessary pre-task questions about the product, industry, or other relevant areas.
Conduct the tasks – All tasks should be designed around predefined goals.
Post-session interview – Ask the questions that weren’t substantial enough to disturb the flow during testing, how they felt the product performed, and what things could have been done better to uncover more about the experience. And always thank the participant for helping with your study.

There are further factors in moderated testing that need careful consideration:

Consider timing and testing schedules – Creating a testing schedule for an extensive pool of participants can be challenging.
Plan how to record participants’ performances and choose your preferred monitoring tools.
Ensure to design and deliver tasks in a set order to maintain a natural flow through testing.

Tips for facilitators moderating usability tests

Ideally, each moderator should be experienced in conducting tests and well acquainted with UX research and the product under scrutiny.

However, the following points offer a few basic essential tips for those planning their own usability testing.

Be clear and concise—keep a steady tone and retain consistency between participants.
Try not to agree or disagree with participant responses. Assure them there is no right or wrong; the value is in their answers and actions.
Record all sessions—there’s great value in recording and transcribing interviews and task conversations.
Take notes or designate a second team member to take them for you.
Give participants enough time and space to deliberate and explore each task.
Stay as quiet as possible during tasks unless it’s necessary to interject.
Know when to interject (and when not to) and the best way to do so without disrupting the process.
Leave the conclusions until the test is over—focus on the task before you.
Monitor closely for physical gestures and changes in expression.
Use personal and relevant data, where possible, to keep the participant engaged and the tasks realistic.
Encourage participants to describe what they’re doing, wherever appropriate.
Practice scripts and become well acquainted with any technology you plan to use. Pilot testing is the ideal opportunity to iron out problems before the real tests begin.
Don’t leave anything to chance—create checklists for each task and stage.
Keep each test on track—even though it’s necessary to explore unpredictable avenues as they happen, following your test program maintains flow and continuity between tasks and participants.

Summary

There are significant differences between unmoderated and moderated usability testing. Choosing the option fit for purpose is down to the design and research team and resources. As with any testing, defining goals around what you hope to achieve is essential, and the methods you use to uncover your data should revolve around them.

When conducted correctly, usability testing uncovers unexpected issues as well as validating ideas and product builds. It’s an essential part of UX research and a valuable resource for examining usability issues, user thoughts, values, behaviours, and improving their experience.

If you would like to know more about this interesting and potentially extremely useful technique, ring us on +44(0)800 024624 or email us at hello@ux247.com.