Abstract
Chatbots powered by large language models (LLMs) are increasingly prevalent in various domains. Nonetheless, they face challenges such as hallucinations and losing context during extended conversations. This study tackles these issues by proposing a multi-agent strategy for chat architecture where multiple LLMs focus on distinct tasks to enhance the quality of their output. The suggested solution involves a supervisor agent working in conjunction with a document search and review module. We assess the performance of information systems with chatbots designed to respond to sustainability questions in English and handle technical documentation for plant equipment in Polish. A comprehensive analysis of commercial and open-source models revealed that Qwen2.5 v14b’s performance is comparable to that of the Gemini family models.
Paper Type
Full Paper
DOI
10.62036/ISD.2025.58
How to moderate LLM based chats from hallucinations?
Chatbots powered by large language models (LLMs) are increasingly prevalent in various domains. Nonetheless, they face challenges such as hallucinations and losing context during extended conversations. This study tackles these issues by proposing a multi-agent strategy for chat architecture where multiple LLMs focus on distinct tasks to enhance the quality of their output. The suggested solution involves a supervisor agent working in conjunction with a document search and review module. We assess the performance of information systems with chatbots designed to respond to sustainability questions in English and handle technical documentation for plant equipment in Polish. A comprehensive analysis of commercial and open-source models revealed that Qwen2.5 v14b’s performance is comparable to that of the Gemini family models.
Recommended Citation
Rurański, J., Nowak, K., Góras, M., Dworak, K. & Nurzyńska, K. (2025). How to moderate LLM based chats from hallucinations?In I. Luković, S. Bjeladinović, B. Delibašić, D. Barać, N. Iivari, E. Insfran, M. Lang, H. Linger, & C. Schneider (Eds.), Empowering the Interdisciplinary Role of ISD in Addressing Contemporary Issues in Digital Transformation: How Data Science and Generative AI Contributes to ISD (ISD2025 Proceedings). Belgrade, Serbia: University of Gdańsk, Department of Business Informatics & University of Belgrade, Faculty of Organizational Sciences. ISBN: 978-83-972632-1-5. https://doi.org/10.62036/ISD.2025.58