Users’ Experiences with Chatbots: Findings from a Questionnaire Study

Abstract

For chatbots to be broadly adopted by users, it is critical that they are experienced as useful and pleasurable. While there is an emerging body of research concerning user uptake and use of chatbots, there is a lack of theoretically grounded studies detailing what constitutes good or poor chatbot user experiences. In this paper, we present findings from a questionnaire study involving more than 200 chatbot users who reported on episodes of chatbot use that they found particularly satisfactory or frustrating. The user reports were analysed with basis in theory on user experience, with particular concern for pragmatic and hedonic attributes. We found that pragmatic attributes such as efficient assistance (positive) and problems with interpretation (negative) were important elements in user reports of satisfactory and frustrating episodes. Hedonic attributes such as entertainment value (positive) and strange and rude responses (negative) were also frequently mentioned. Older participants tended to report on pragmatic attributes more often, whereas younger participants tended to report on hedonic attributes more often. Drawing on the findings, we propose four high-level lessons learnt that may benefit chatbot service providers, and we suggest relevant future research.