PACIS 2022 Proceedings
Paper Number
1115
Abstract
This study aimed to explore and compare the performance of chatbots and human agents on different tasks in online customer services. With practical Twitter data from several customer service accounts, we categorized the users’ inquiries into three categories: uncertain, complex, and ambiguous, and manually evaluated the responding posts in metrics of appropriateness, helpfulness, and empathy. Our results show that, although human agents outperformed chatbots in most tasks under different metrics, chatbots had similar performance in providing helpful responses to uncertain inquiries with a shorter response time. These results indicate that chatbots have relative advantages in responding to uncertain tasks. The findings of our study provide practical implications for chatbot design that, to minimize total transaction costs, organizations may want to consider including a module to redirect uncertain inquiries to human agents after identifying the type of the inquiry.
Recommended Citation
Yang, Chi-Heng; Yen, Ju-Chun; and Wang, Tawei (David), "Human vs machine: Do customer service chatbots perform better?" (2022). PACIS 2022 Proceedings. 61.
https://aisel.aisnet.org/pacis2022/61
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
Paper Number 1115