Paper Number
ECIS2026-1771
Paper Type
CRP
Abstract
Artificial Intelligence (AI) is increasingly deployed in high-stakes domains such as healthcare, where decisions carry significant consequences for individuals and society. This amplifies the need for safe and trustworthy AI systems. While traditional explainable AI (XAI) methods aim to increase transparency, they often omit the uncertainties that are inherent in XAI-augmented decision-making, where AI supports human decision-makers. To better understand the sources and effects of uncertainty in XAI-augmented decision-making and structure the existing body of knowledge, we conduct a structured literature review focused on four main sources of uncertainty: data uncertainty, model uncertainty, XAI method uncertainty, and human uncertainty. Our review shows that these sources are largely examined in isolation, resulting in a fragmented understanding of how uncertainties interact and influence decision-making. Based on these findings, we outline how multiple sources of uncertainty can be incorporated into uncertainty-aware explanations and propose an integrated research agenda for developing uncertainty-aware XAI.
Recommended Citation
Förster, Maximilian; Hagn, Michael; Hambauer, Nico; Jaki, Paula Kathrin Viktoria; Obermeier, Andreas Alexander; Schauer, Andreas; and Schiller, Alexander, "Understanding Uncertainties In Explainable Ai: A Structured Literature Review And Research Agenda" (2026). ECIS 2026 Proceedings. 8.
https://aisel.aisnet.org/ecis2026/bus_analytics/bus_analytics/8
Understanding Uncertainties In Explainable Ai: A Structured Literature Review And Research Agenda
Artificial Intelligence (AI) is increasingly deployed in high-stakes domains such as healthcare, where decisions carry significant consequences for individuals and society. This amplifies the need for safe and trustworthy AI systems. While traditional explainable AI (XAI) methods aim to increase transparency, they often omit the uncertainties that are inherent in XAI-augmented decision-making, where AI supports human decision-makers. To better understand the sources and effects of uncertainty in XAI-augmented decision-making and structure the existing body of knowledge, we conduct a structured literature review focused on four main sources of uncertainty: data uncertainty, model uncertainty, XAI method uncertainty, and human uncertainty. Our review shows that these sources are largely examined in isolation, resulting in a fragmented understanding of how uncertainties interact and influence decision-making. Based on these findings, we outline how multiple sources of uncertainty can be incorporated into uncertainty-aware explanations and propose an integrated research agenda for developing uncertainty-aware XAI.