Paper Type
ERF
Abstract
AI-based clinical decision support systems are increasingly integrated into diagnostic workflows, yet their impact depends not only on algorithmic accuracy but also on how clinicians respond to AI recommendations. The challenge becomes especially salient under human–AI disagreement, when clinicians must choose between conflicting judgments without immediate feedback. Drawing on metacognitive regulation theory, we frame disagreement as a problem of monitoring and control and develop a process model linking AI interface design to diagnostic outcomes. We focus on two interface features, explainable AI and cognitive forcing functions, and argue that they influence distinct stages of regulation. We propose a controlled experiment to examine how these features shape monitoring, control, diagnostic regulation behaviors, and diagnostic accuracy. By viewing AI-assisted diagnosis as a regulatory process rather than solely a trust decision, this study explains how interface design can foster appropriate reliance in clinical practice.
Paper Number
1144
Recommended Citation
Farahmandazad, Dorsa and Yoo, Chul Woo, "From Confidence to Checking: How Interfaces Shape Appropriate Reliance on Clinical AI" (2026). AMCIS 2026 Proceedings. 1.
https://aisel.aisnet.org/amcis2026/ai_systdesign/ai_systdesign/1
From Confidence to Checking: How Interfaces Shape Appropriate Reliance on Clinical AI
AI-based clinical decision support systems are increasingly integrated into diagnostic workflows, yet their impact depends not only on algorithmic accuracy but also on how clinicians respond to AI recommendations. The challenge becomes especially salient under human–AI disagreement, when clinicians must choose between conflicting judgments without immediate feedback. Drawing on metacognitive regulation theory, we frame disagreement as a problem of monitoring and control and develop a process model linking AI interface design to diagnostic outcomes. We focus on two interface features, explainable AI and cognitive forcing functions, and argue that they influence distinct stages of regulation. We propose a controlled experiment to examine how these features shape monitoring, control, diagnostic regulation behaviors, and diagnostic accuracy. By viewing AI-assisted diagnosis as a regulatory process rather than solely a trust decision, this study explains how interface design can foster appropriate reliance in clinical practice.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
AI SYSTEM