Paper Number
ICIS2025-2819
Paper Type
Complete
Abstract
Artificial intelligence (AI)-based decision-making systems have been shown to outperform humans. However, in critical decision-making domains like healthcare, human decision-makers often mistrust and are reluctant to follow the recommendations of black-box AI systems because they perceive the system to be biased. This study aims to advance AI for social good by illuminating the mechanisms by which perceived bias in AI systems affects users’ decision-making effectiveness and how different explanation types mitigate these effects. Drawing from dual-process theory and the theory of effective use, we propose two mechanisms that mediate the effects of perceived system bias in explainable AI systems: a cognitive mechanism of learning and an emotional mechanism of anticipated regret. Our study found that the cognitive mechanism of learning primarily mediates the relationship between perceived system bias and decision-making effectiveness, and feature importance explanations mitigate the negative effects of perceived system bias more effectively than counterfactual explanations.
Recommended Citation
Negi, Kartikeya; Werder, Karl; Zhang, Rongen "Sophia"; and Ramesh, Balasubramaniam, "Impact of the Perceived System Bias and Type of AI Explanations on Decision-Making Effectiveness in Explainable AI Systems: Cognitive and Emotional Mechanisms" (2025). ICIS 2025 Proceedings. 21.
https://aisel.aisnet.org/icis2025/is_good/is_good/21
Impact of the Perceived System Bias and Type of AI Explanations on Decision-Making Effectiveness in Explainable AI Systems: Cognitive and Emotional Mechanisms
Artificial intelligence (AI)-based decision-making systems have been shown to outperform humans. However, in critical decision-making domains like healthcare, human decision-makers often mistrust and are reluctant to follow the recommendations of black-box AI systems because they perceive the system to be biased. This study aims to advance AI for social good by illuminating the mechanisms by which perceived bias in AI systems affects users’ decision-making effectiveness and how different explanation types mitigate these effects. Drawing from dual-process theory and the theory of effective use, we propose two mechanisms that mediate the effects of perceived system bias in explainable AI systems: a cognitive mechanism of learning and an emotional mechanism of anticipated regret. Our study found that the cognitive mechanism of learning primarily mediates the relationship between perceived system bias and decision-making effectiveness, and feature importance explanations mitigate the negative effects of perceived system bias more effectively than counterfactual explanations.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
06-SocialGood