Abstract

Abstract Agentic AI systems — where multiple AI agents work together automatically to complete complex tasks — are being rapidly adopted by organizations. Unlike simple AI tools that respond to a single question or command, these systems make a series of decisions on their own, often without human involvement at each step. This makes it difficult for employees to understand how a final decision was reached, who or what is responsible if something goes wrong, and whether the output can be trusted. As organizations begin using these systems for important tasks such as financial forecasting, supply chain management, and hiring decisions, a key question arises: how do employees decide how much to trust an AI system whose internal reasoning they cannot see or fully understand? This TREO Talk proposes a research agenda on Trust Calibration in Agentic AI Systems (TCAS) — focusing on how people form, adjust, and sometimes misplace their trust when working with autonomous AI pipelines. Building on IS research on automation bias (Parasuraman & Manzey, 2010), technology trust (McKnight et al., 2002), and algorithm aversion (Dietvorst et al., 2015), this study argues that multi-agent AI creates trust challenges that existing theories do not fully explain. These systems make it hard to identify which agent caused an error, typically show only the final result rather than the step-by-step reasoning, and change their behavior over time — all of which make it difficult for users to develop a stable and reliable sense of trust. This talk raises three research questions: (1) How do employees build mental models of multi-agent AI systems, and how do these models affect how much they trust the system? (2) Under what conditions do workers trust these systems too much (automation bias) or too little (algorithm aversion)? (3) What organizational practices — such as explainability features, human approval checkpoints, or audit logs — can help employees develop an appropriate level of trust? To explore these questions, this study proposes a mixed-methods approach: scenario-based experiments with organizational employees to measure trust miscalibration, combined with in-depth interviews with early users of agentic AI platforms such as Microsoft Copilot Studio and LangGraph. This research sits at the intersection of human-computer interaction, IT governance, and organizational behavior — all central themes in IS research. The talk invites community feedback on the theoretical framework, research design, and broader implications. As organizations increasingly rely on autonomous AI pipelines for consequential decisions, understanding how employees calibrate their trust in these systems is both an urgent research challenge and a practical concern for businesses, regulators, and technology designers. References Dietvorst, B. J., Logg, J. M., & Logg, J. M. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. McKnight, D. H., Choudhury, V., & Kacmar, C. (2002). Developing and validating trust measures for e-commerce: An integrative typology. Information Systems Research, 13(3), 334–359. Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410.

Download

Share

COinS