There are many reasons why it is valuable to run pilot sessions for design research. But when the project is international design research, and by that I mean the research is taking place in:
- different languages
- different time zones
the value in a pilot rises exponentially. That is also true when you plan to run international design research simultaneously in more than one market.
What is a pilot session?
A pilot session provides the opportunity to test the discussion guide (and tasks if usability testing) prior to running the full research. It takes place 2 to 4 days ahead of the main research, dependent on timezone. It can also be used to identify problems with a prototype if usability testing, although additional time may need to be allowed for changes.
The pilot is conducted with a participant or multiple participants if the discussion guide is different for specific personas. Participants are recruited with identical profiles to the full research and generally recruited at the same time. From the participant perspective, they would not be able to tell the difference between the pilot and a real session.
The tech set up is identical to the full test, other than if face to face research is taking place, rather than remote. That is because it may not be economically viable to use a research facility for a one-off session.
What pilot sessions tell us
Discussion guides contains the questions and structure the moderator will use to guide the conversation with the participant. During the planning stage the discussion guide will be drafted to cover every area the client is interested in and structured around sections, such as:
- warm up
- background and profile
- recent online experience
- etc.
Each section will have an approximate time-frame assigned. Warm up may be 5 minutes, background and profile 10 minutes and so on. The full scope of the discussion guide may not be known when participant recruitment takes place and so time may be a constraint. A typical research session lasts anywhere between 60 and 120 minutes.
The purpose of the pilot session is to ensure that the full sessions generate the insight required. The pilot will feedback on:
- Priorities – is the timing right between the different areas of questioning? If time is short where do we cut back?
- Depth – does the discussion guide generate the insight expected? Do we need more focus in a specific area?
- Structure – does the order work? Would it be better to move a section back, bring a section forward?
- Assets – are the research assets we are using working? Do we need to change them?
How much cost does a pilot add to international design research?
It shouldn’t add much additional cost to the project to run a pilot session. It is all about the planning. We make it part of the briefing and planning process and include the recruitment as part of the brief to the recruiter. The cost buckets and amounts are broadly as follows:
- additional consultant time = 1-2 hours
- additional participant cost = $100-200
- Simultaneous translator = $150-300
That should be everything. The revisions to the discussion guide should all be carried out as part of the planning process. When we run projects that include pilots we allow 2 1/2 days for the moderator that includes the briefing call, creation of discussion guide, pilot session, revision of discussion guide, immersion and final prep. For all our projects the consultants are supported by Research Ops so a lot of the logistics is taken away from them so they can focus where they add the most value.
Why pilot sessions are more valuable for international Design Research
Different timezones
We have already established that pilot sessions are of value by ensuring that the discussion guide achieves the goals before the research “proper” begins. Pilot sessions allow you to have confidence in the research approach and moderator in a way that observing sessions and offering feedback between them doesn’t.
Feedback between sessions is hard to deliver and hard to take. Remote research makes this even harder and doesn’t help make the research more effective.
The researcher may already be stressed and the last thing they need is to feel their capability as a moderator is being criticized – even if it isn’t. Minor tweaks are OK but wholesale changes are very hard to make work effectively and in any case, you are losing sessions as you make those changes.
For the client, it can feel like the moderator isn’t listening to the feedback. The moderator will have spent a couple of days really getting to grips with the discussion guide and last minute changes don’t always stick. The client can become very frustrated about going over the same thing between every session.
A pilot session removes all this pain. Because feedback is being delivered in the context of developing the discussion guide it isn’t taken as personal. It can also be worked on, fine tuned and then added so is included in the final immersion. That makes sure the moderator knows exactly what they are doing and you are confident in them.
Anyone who has watched international design research with an 8-hour time difference knows, you struggle to bring your a-game. Sometimes you can’t watch at all. Sometimes your team doesn’t show up, or the connection is poor. There are a whole host of reasons why you should plan to not be there and a pilot session is the primary solution.
Different languages
When the research language is different to the client teams, we use a simultaneous translator. Their role is to listen in the research language (say Chinese) and speak out loud in the audience language (say English). Simultaneous translators like to work in teams of two because it is hard work. If you are lucky, you will find one happy to work alone.
You pay for simultaneous translators whether there is something to translate or not. if you book them for five, sixty minute sessions in a day, that is what you will pay for plus some set up time. In the event of a no-show you are still paying and you will also pay for the participant that is scheduled in later to replace them. Costs can soon mount up with additions of $2-300 per participant.
If you don’t view the research live you can ask your simultaneous translator to translate from the video files. For sure this means you will be watching them a day or two behind live, but it will mean you only pay for the translation of the sessions that actually run. The translator also does a better job because they can do the work away from the staring eyes of the audience.
Summary
If you are not already using pilot sessions to make your international design research better I urge you to reconsider. For very little additional outlay, if properly planned, they can deliver a huge amount of value. They save money, avoid wasting sessions, ensure objectives are met and that the moderator is very clear on what is required by the client. That is the case for single market research projects but even more so for international design research.
If you would like to know more about how to use pilot sessions to enhance your international design research then please, get in touch on +44(0)800 024624 or email us at hello@ux247.com.
