International Expert Review of AI chatbot
MSG.AI uses expert review to train AI Chatbot in UK and Mexico
MSG.AI is an American artificial intelligence (AI) company and developer of human-computer interaction technologies. Founded in 2015, their aim is to make contacting companies a brilliant customer service experience. They empower AI to respond to customer issues that are repeatable, while enabling human agents to focus on high impact work.
UX24/7 was commissioned by MSG.AI to undertake an expert evaluation on a Nescafe Dolce Gusto chatbot that operates on Facebook Messenger. The focus was on the following areas:
- Occurrence of the planned responses, such as the reaction when the bot does not understand the user
- Responses from the bot that the user doesn’t think are correct, when the bot thinks it is correct
- Duty of care and trolling, such as how the bot handles bad language, threatening behaviour, abuse, self-harm etc.
Expert reviews were done on a Spanish version of the chatbot for the Mexican market as well as an English version.
An expert review is a detailed evaluation of a chatbots user experience (UX) operation and is carried out by a UX consultant. Detailed UX issues are identified following a walk-through of key user journeys and interactions through natural language.
The UX consultant identifies key user journeys for the chatbot (i.e. tasks that are commonly carried out by users when interacting with the chatbot) and any other scenarios that need to be undertaken to evaluate the interaction with the chatbot.
A cognitive walkthrough of each user journey/scenario is undertaken and UX issues are identified. This is where the consultant adopts the mindset of a user and recreates, as closely as possible, the type of real-world interactions that would be expected. The interaction is evaluated, and issues are recorded using severity ratings, that impact the user experience.
These specific areas were explored by the consultants:
- Navigation – how effectively can the bot offer choices to the user in order to speed up responses
- Content – the accuracy and completeness of responses provided by a bot, and when in context would it meet the user’s information needs
- Presentation – testing the tone of voice and whether the responses given by the bot are in line with expected behaviour/brand image
- Function – testing the efficiency with which a bot solves a user request, the goal being to use the least number of questions to provide the answer
- Feedback – measuring the effectiveness of recommendations a bot provides in case of an error
- Personalisation – whether the bot can remember historical interactions, and the user’s preferences, to improve it’s understanding of current context
- Evolution – does the bot’s intelligence increase over time? This is a characteristic of the bot technology that can be assessed over a period of time, and through regular interaction.
QUICK FACTS
- Expert review against structured framework
- Chatbot user journeys evaluated
- Mexican Spanish & English versions evaluated in market
- Chatbot effectiveness and Governance
The consultants found that the language used by the chatbot was mostly clear and appropriate. It had a friendly and informal tone but was sometimes over-wordy. Help advice was well worded and assisted by showing diagrams. This was useful and allowed the user to have control of the process.
The chatbot often failed to recognise key words e.g. ‘capsules’, ‘spare parts’ and escalated problems rather than providing standard help content to questions that users are likely to ask. When problems arose, the chatbot didn’t direct the user to an advisor or agent who may be able to help in the situation.
The product information provided on the Nescafe Dolce Gusto range didn’t help to guide the user on product selection. The language used leant towards ‘marketing speak’ rather than a concise summary of features which the user is likely to want.
However, the chatbot handled inappropriate language and questions effectively, quickly detecting and stopping the use of obscene language.
These are some of the key areas of insight we uncovered for MSG.AI:
- The interactive experience can be enhanced by providing step-by-step instructions that allow the users to execute actions on their machines and provide feedback to the chatbot at each step.
- Relying on the use of animated images, without numbered steps can be confusing and inefficient as many of the scenarios are complex.
- To aid product selection, it was recommended that the chatbot ‘funnels’ the user to a specific product based on a sequence of multiple-choice questions.
- The user experience of the Spanish version was similar to the experience with the English one but some key experience elements did not exist on the Spanish version.
- The Spanish version of the chatbot has some text errors and still had some elements in English so better localisation was required.
Our consultants found that there was an opportunity to create a more natural conversational scenario for the Nescafe chatbot.
Many of the observations about communication and empathy are related to either saying too much or too little during an interactive dialogue which results in keeping the interaction quite artificial.
Less predefined paths and more step-by-step elements of conversation can lead to a better and more natural communication.
