Paper Number
ICIS2026-2832
Paper Type
Completed
Abstract
Generative AI is increasingly embedded in knowledge work, requiring users to verify fluent but error-prone outputs. We report a preregistered between-subjects experiment with 280 participants from a university-centered convenience sample who verified three AI-generated summaries containing planted surface, logic, and hallucination errors. Participants received standard usage or one of three cognitive-forcing implementations: a 20-second delay, pre-AI source notes, or a post-exposure category checklist. No intervention detectably improved error detection in this continuous-text task; the pooled contrast was near zero (d = −0.033, 95% CI [−0.293, 0.226]). Checklist and Decision First increased task time, while Checklist increased self-reported critical evaluation; yet 51.3% of error-linked checklist flags were not accompanied by error correction. These findings establish, within this task, that oversight and effective correction are empirically separable outcomes. They document an oversight-correction gap and motivate task intervention fit as a candidate boundary condition for cognitive forcing functions.
Recommended Citation
Kelch, Yngve, "Fluent but False: The Oversight-Correction Gap in Generative AI Verification" (2026). ICIS 2026 Proceedings. 24.
https://aisel.aisnet.org/icis2026/user_engage/user_engage/24
Fluent but False: The Oversight-Correction Gap in Generative AI Verification
Generative AI is increasingly embedded in knowledge work, requiring users to verify fluent but error-prone outputs. We report a preregistered between-subjects experiment with 280 participants from a university-centered convenience sample who verified three AI-generated summaries containing planted surface, logic, and hallucination errors. Participants received standard usage or one of three cognitive-forcing implementations: a 20-second delay, pre-AI source notes, or a post-exposure category checklist. No intervention detectably improved error detection in this continuous-text task; the pooled contrast was near zero (d = −0.033, 95% CI [−0.293, 0.226]). Checklist and Decision First increased task time, while Checklist increased self-reported critical evaluation; yet 51.3% of error-linked checklist flags were not accompanied by error correction. These findings establish, within this task, that oversight and effective correction are empirically separable outcomes. They document an oversight-correction gap and motivate task intervention fit as a candidate boundary condition for cognitive forcing functions.
Comments
14_User_behavior