Distorted Deliberation: A Field Study of Metacognitive Miscalibration in AI-Assisted Legal Education
Paper Number
ICIS2026-2261
Paper Type
Completed
Abstract
The human-in-the-loop paradigm relies on accurate oversight to safely integrate generative AI into professional workflows. This study investigates a critical vulnerability in this assumption: metacognitive miscalibration, the divergence between users' perceived and actual alignment with AI-generated reasoning. In a field experiment, students and an LLM reasoned from identical legal cases. We anchored students' deliberation using the IRAC framework, then benchmarked their evaluations of the AI's output against an objective semantic measure. Panel data analysis reveals miscalibration is dynamically induced by task complexity, producing "distorted deliberation" where evaluative accuracy collapses despite structured engagement. Students competently detect errors in simple tasks, but this competence breaks down under analytical depth. Furthermore, recognizing novel AI insights remains a persistent blind spot amplified by case complexity. These findings reframe miscalibration as a predictable breakdown, demonstrating that procedural oversight is an insufficient safeguard in high-stakes domains.
Recommended Citation
Tan, Tristan Jin Hau and Wang, Qiuhong, "Distorted Deliberation: A Field Study of Metacognitive Miscalibration in AI-Assisted Legal Education" (2026). ICIS 2026 Proceedings. 7.
https://aisel.aisnet.org/icis2026/learn/learn/7
Distorted Deliberation: A Field Study of Metacognitive Miscalibration in AI-Assisted Legal Education
The human-in-the-loop paradigm relies on accurate oversight to safely integrate generative AI into professional workflows. This study investigates a critical vulnerability in this assumption: metacognitive miscalibration, the divergence between users' perceived and actual alignment with AI-generated reasoning. In a field experiment, students and an LLM reasoned from identical legal cases. We anchored students' deliberation using the IRAC framework, then benchmarked their evaluations of the AI's output against an objective semantic measure. Panel data analysis reveals miscalibration is dynamically induced by task complexity, producing "distorted deliberation" where evaluative accuracy collapses despite structured engagement. Students competently detect errors in simple tasks, but this competence breaks down under analytical depth. Furthermore, recognizing novel AI insights remains a persistent blind spot amplified by case complexity. These findings reframe miscalibration as a predictable breakdown, demonstrating that procedural oversight is an insufficient safeguard in high-stakes domains.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
22_Digital_learning