Paper Type
ERF
Abstract
As AI becomes increasingly integrated into high-stakes decision-making, ensuring appropriate reliance—users’ ability to calibrate their trust based on AI reliability—remains a critical challenge. Metacognitive AI, which monitors and regulates its own decision-making, has the potential to improve trust calibration by fostering trust resilience and critical thinking engagement. However, its enhanced self-reflective capabilities may also lead to over-reliance due to its perceived authority. Drawing on dual-process theory and obedience to authority theory, this study investigates how metacognitive AI influences user reliance behaviors. Using a between-subjects experimental design, 200 participants will interact with AI advisors exhibiting high or low metacognitive ability in a medical diagnosis task. We examine the effects on trust resilience, critical thinking, and reliance patterns, moderated by Need for Cognition (NFC). Findings will contribute to the design of AI systems that foster appropriate reliance and decision-making autonomy, reducing automation bias and improving human-AI collaboration.
Paper Number
2135
Recommended Citation
Mehri, Bita and Ghasemaghaei, Maryam, "The Impact of Metacognitive AI on Appropriate Reliance in AI-Assisted Decision-Making: The Role of Trust Resilience and Critical Thinking" (2025). AMCIS 2025 Proceedings. 27.
https://aisel.aisnet.org/amcis2025/sig_odis/sig_odis/27
The Impact of Metacognitive AI on Appropriate Reliance in AI-Assisted Decision-Making: The Role of Trust Resilience and Critical Thinking
As AI becomes increasingly integrated into high-stakes decision-making, ensuring appropriate reliance—users’ ability to calibrate their trust based on AI reliability—remains a critical challenge. Metacognitive AI, which monitors and regulates its own decision-making, has the potential to improve trust calibration by fostering trust resilience and critical thinking engagement. However, its enhanced self-reflective capabilities may also lead to over-reliance due to its perceived authority. Drawing on dual-process theory and obedience to authority theory, this study investigates how metacognitive AI influences user reliance behaviors. Using a between-subjects experimental design, 200 participants will interact with AI advisors exhibiting high or low metacognitive ability in a medical diagnosis task. We examine the effects on trust resilience, critical thinking, and reliance patterns, moderated by Need for Cognition (NFC). Findings will contribute to the design of AI systems that foster appropriate reliance and decision-making autonomy, reducing automation bias and improving human-AI collaboration.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIGODIS