Paper Number

ICIS2026-2261

Paper Type

Completed

Abstract

The human-in-the-loop paradigm relies on accurate oversight to safely integrate generative AI into professional workflows. This study investigates a critical vulnerability in this assumption: metacognitive miscalibration, the divergence between users' perceived and actual alignment with AI-generated reasoning. In a field experiment, students and an LLM reasoned from identical legal cases. We anchored students' deliberation using the IRAC framework, then benchmarked their evaluations of the AI's output against an objective semantic measure. Panel data analysis reveals miscalibration is dynamically induced by task complexity, producing "distorted deliberation" where evaluative accuracy collapses despite structured engagement. Students competently detect errors in simple tasks, but this competence breaks down under analytical depth. Furthermore, recognizing novel AI insights remains a persistent blind spot amplified by case complexity. These findings reframe miscalibration as a predictable breakdown, demonstrating that procedural oversight is an insufficient safeguard in high-stakes domains.

Comments

22_Digital_learning

Share

COinS
 
Dec 14th, 12:00 AM

Distorted Deliberation: A Field Study of Metacognitive Miscalibration in AI-Assisted Legal Education

The human-in-the-loop paradigm relies on accurate oversight to safely integrate generative AI into professional workflows. This study investigates a critical vulnerability in this assumption: metacognitive miscalibration, the divergence between users' perceived and actual alignment with AI-generated reasoning. In a field experiment, students and an LLM reasoned from identical legal cases. We anchored students' deliberation using the IRAC framework, then benchmarked their evaluations of the AI's output against an objective semantic measure. Panel data analysis reveals miscalibration is dynamically induced by task complexity, producing "distorted deliberation" where evaluative accuracy collapses despite structured engagement. Students competently detect errors in simple tasks, but this competence breaks down under analytical depth. Furthermore, recognizing novel AI insights remains a persistent blind spot amplified by case complexity. These findings reframe miscalibration as a predictable breakdown, demonstrating that procedural oversight is an insufficient safeguard in high-stakes domains.

When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.