The methodology for conducting quantitative research (online task studies) of this type involves the following:
- A panel of respondents segmented to as close as possible the target profiles specified by the client
- A study design including tasks, questions and any study logic
- Software for delivering the study and the time required to code it
- Analysis of the data set and reporting of findings and recommendations
Panel Size and Selection:
The main reasons for running quantitative design research are that users are geographically spread out, or that a statistically relevant sample size is required.
With international studies, where statistical relevance is not required, smaller numbers can be sufficient – for example if there are ten countries we may only wish to research with 20 in each. With this type of research, we are using the quantitative methodology as a proxy for qualitative user testing most likely because it is more cost effective than to run the research in each of ten countries.
In a single country study we may wish to use a panel of 200 minimum in any case because we are using the methodology because we need a high volume of responses to give us statistical relevance.
In both cases it may make sense to run studies with hundreds of users because there is not much more effort in analysing large or small data sets so it is mainly a cost question. The only time this isn’t the case is when a study includes lost of questions that allow respondents to give unique answers – i.e. they type a sentence about what they liked or didn’t like. Each of these individual responses needs to be read and as this is time consuming it increases the cost of analysis.
The study is delivered to the panel using a link that is emailed which makes it very easy to course respondents from any one of a number of panel providers, or even the clients’ internal customer panel. We are flexible with where the respondents come from and it is simply a case of matching budget and profiles.
Study Design and Questions:
Once an online study goes live it can’t be changed – it is a fundamental difference between qualitative research, which can be adapted as you go, and quantitative research, and so the study design must be perfect when it is launched.
To achieve this we use the following process:
- Hold a briefing meeting or call to discuss in detail the requirement, scope of tasks and questions to be asked.
- Draft a study script in a word document format and share this with the client.
- Iterate and improve until we are all happy with the draft script.
- Code the script into the study software and carry out a test run.
- When we are happy with the study send a test link to the client for test and sign-off or further iteration.
- Go live only when the final test link is signed off.
During the briefing meeting or call we also ask about the reporting format, the type of tables and charts you are expecting to see and cuts of data. For example if you are expecting to see a gender split we need to make sure gender is a question being asked.
We are able to use a range of technologies to run online task studies including UserZoom and Loop11. There are strengths and weaknesses with many of the software products available and we are happy to recommend the best product for your requirement. If you use an in house tool or have a licence already in place we are happy to use that.
When the study is coded and the test link signed off the study will be made live. We will then monitor the completion rate and liaise with the panel providers to ensure panellists are being sent the study and that we are achieving the quota.
Analysis and Reporting:
When the quota has been achieved the study will be turned off and analysis begins. Quantitative studies tend to create very large data sets with 100’s of responses and may require statistical analysis to answer some of the research questions.
Some of the study software tools come with their own built in visualisation capabilities that can quickly and easily carry out cross tabulation and provide tables and charts. Where this is not the case, or where the research requires more detailed analysis we may export the data set into excel or a more powerful package such as SPSS.
Have a Project in Mind?