Paper Number
ECIS2026-2455
Paper Type
CRP
Abstract
Retrieval-augmented generation (RAG) architecture enhances large language models (LLMs) by incorporating external, domain-specific data sources, thereby reducing hallucinations and improving response accuracy. This approach departs from purely generative systems and offers a scalable, cost-efficient alternative for enhancing LLMs. However, the construction of ground truth to enable evolvability (ability to integrate new data continuously) and adaptability (ability to adjust responses to user context and intents dynamically) remains challenging. This study investigates how the challenges of establishing ground truth manifest at the organizational, unit, team, and individual levels within a countywide Swedish healthcare organization. Our analysis reveals that the main challenges stem from the heterogeneity of knowledge practices, which complicate the creation of reliable retrieval from the knowledge base. Furthermore, we identify a trade-off between attempts to enhance accuracy and maintain contextual relevance. These findings contribute practical insights into the micro-foundations of evolvability and adaptability in RAG-based chatbot development.
Recommended Citation
Alaie, Parisima; Vallgren, Erik Johannes; and Sandberg, Johan, "Ground Truth Heterogeneity: Exploring Development and Non-Clinical Use Of Retrieval-Augmented Generation Chatbot In Healthcare" (2026). ECIS 2026 Proceedings. 16.
https://aisel.aisnet.org/ecis2026/genai/genai/16
Ground Truth Heterogeneity: Exploring Development and Non-Clinical Use Of Retrieval-Augmented Generation Chatbot In Healthcare
Retrieval-augmented generation (RAG) architecture enhances large language models (LLMs) by incorporating external, domain-specific data sources, thereby reducing hallucinations and improving response accuracy. This approach departs from purely generative systems and offers a scalable, cost-efficient alternative for enhancing LLMs. However, the construction of ground truth to enable evolvability (ability to integrate new data continuously) and adaptability (ability to adjust responses to user context and intents dynamically) remains challenging. This study investigates how the challenges of establishing ground truth manifest at the organizational, unit, team, and individual levels within a countywide Swedish healthcare organization. Our analysis reveals that the main challenges stem from the heterogeneity of knowledge practices, which complicate the creation of reliable retrieval from the knowledge base. Furthermore, we identify a trade-off between attempts to enhance accuracy and maintain contextual relevance. These findings contribute practical insights into the micro-foundations of evolvability and adaptability in RAG-based chatbot development.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.