Paper Number
ECIS2026-2353
Paper Type
CRP
Abstract
AI systems increasingly support decisions, yet users frequently override recommendations. To address this, research suggests providing explanations to increase trust and adherence. However, users still override recommendations even when explanations are provided. We study why this occurs through a design science study with a provider of a hotel pricing system. We identify two factors driving excessive overrides: misaligned fairness perceptions and lack of consequence awareness. Drawing on Anchoring and Adjustment Heuristic and Social Comparison Theory, we develop two design principles: (1) Fairness Calibration provides reference points like competitor prices to align fairness judgments with market realities, and (2) Consequence Understanding quantifies potential revenue losses to clarify override impacts. We evaluate a prototype through think-aloud studies with hotel managers. Results demonstrate user acceptance and adherence intention. Together, our findings highlight that effective human-AI collaboration requires addressing also the decision context, offering actionable design knowledge for practitioners and expanding XAI research.
Recommended Citation
Hendriks, Daniel; Holstein, Joshua; Völger, Lukas; Spitzer, Philipp; and Satzger, Gerhard, "Thinking About The Consequences: Design Knowledge For Adherence On Pricing Systems" (2026). ECIS 2026 Proceedings. 27.
https://aisel.aisnet.org/ecis2026/cog_hbis/cog_hbis/27
Thinking About The Consequences: Design Knowledge For Adherence On Pricing Systems
AI systems increasingly support decisions, yet users frequently override recommendations. To address this, research suggests providing explanations to increase trust and adherence. However, users still override recommendations even when explanations are provided. We study why this occurs through a design science study with a provider of a hotel pricing system. We identify two factors driving excessive overrides: misaligned fairness perceptions and lack of consequence awareness. Drawing on Anchoring and Adjustment Heuristic and Social Comparison Theory, we develop two design principles: (1) Fairness Calibration provides reference points like competitor prices to align fairness judgments with market realities, and (2) Consequence Understanding quantifies potential revenue losses to clarify override impacts. We evaluate a prototype through think-aloud studies with hotel managers. Results demonstrate user acceptance and adherence intention. Together, our findings highlight that effective human-AI collaboration requires addressing also the decision context, offering actionable design knowledge for practitioners and expanding XAI research.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.