Abstract
Artificial Intelligence (AI) has accelerated digital transformation across industries, with Large Language Models (LLMs) powering content generation, summarization, and dialogue systems, yet struggling with domain-specific knowledge. In industrial settings, Retrieval-Augmented Generation (RAG) architectures, often implemented as chatbots, address this issue by grounding responses in internal company knowledge. Despite increased industrial deployment, user experience evaluations of RAG-based chatbots remain limited, particularly regarding their effectiveness in supporting domain-specific workplace tasks. In this paper, we present a user evaluation of an RAG-based chatbot conducted in a medium-sized company. Employee feedback on chatbot usability and acceptance is analyzed to guide digitalization efforts in future AI-assisted enterprises.
Paper Type
Poster
DOI
10.62036/ISD.2025.49
Evaluation of User Experience with RAG-based Chatbots for Searching Documentation: Industrial Case Study
Artificial Intelligence (AI) has accelerated digital transformation across industries, with Large Language Models (LLMs) powering content generation, summarization, and dialogue systems, yet struggling with domain-specific knowledge. In industrial settings, Retrieval-Augmented Generation (RAG) architectures, often implemented as chatbots, address this issue by grounding responses in internal company knowledge. Despite increased industrial deployment, user experience evaluations of RAG-based chatbots remain limited, particularly regarding their effectiveness in supporting domain-specific workplace tasks. In this paper, we present a user evaluation of an RAG-based chatbot conducted in a medium-sized company. Employee feedback on chatbot usability and acceptance is analyzed to guide digitalization efforts in future AI-assisted enterprises.
Recommended Citation
Vještica, M., Akik, E., Dimitrieski, V., Hinterleitner, L., Erić, J., Weidenfelder, F.C. & Pisarić, M. (2025). Evaluation of User Experience with RAG-based Chatbots for Searching Documentation: Industrial Case StudyIn I. Luković, S. Bjeladinović, B. Delibašić, D. Barać, N. Iivari, E. Insfran, M. Lang, H. Linger, & C. Schneider (Eds.), Empowering the Interdisciplinary Role of ISD in Addressing Contemporary Issues in Digital Transformation: How Data Science and Generative AI Contributes to ISD (ISD2025 Proceedings). Belgrade, Serbia: University of Gdańsk, Department of Business Informatics & University of Belgrade, Faculty of Organizational Sciences. ISBN: 978-83-972632-1-5. https://doi.org/10.62036/ISD.2025.49