Paper Type
ERF
Abstract
Generative artificial intelligence (GAI) is reshaping cybersecurity by enabling increasingly sophisticated attacks such as AI-generated phishing emails and deepfake videos. While prior research emphasizes technological defenses and adversarial risks, less is known about how routine AI use influences individuals’ ability to detect AI-generated deception. Drawing on cognitive offloading theory and media richness theory, this study examines how AI workflow assimilation, distinguishing between automation and augmentation, affects deception detection capability. We propose that automation, characterized by substitutive AI use, reduces cognitive vigilance and impairs detection accuracy, whereas augmentation, involving complementary human–AI collaboration, enhances analytical engagement and improves detection performance. Further, we argue that these effects are moderated by media richness, with stronger impacts under high-richness threats such as deepfake videos compared to text-based phishing. This study will advance a socio-cognitive perspective on AI-enabled cybersecurity vulnerabilities and contribute to understanding the human consequences of generative AI adoption.
Paper Number
1555
Recommended Citation
Shukla, Akanksha; Nguyen, Thuan; Danjuma, Hannatu; Coulon, Gifty; Olatunde-Thorpe, Jennifer; and Olorunfemi, Taiye Moses, "The Impacts of Generative AI on Individual Ability to Detect AI-Generated Deceptions" (2026). AMCIS 2026 Proceedings. 25.
https://aisel.aisnet.org/amcis2026/sig_sec/sig_sec/25
The Impacts of Generative AI on Individual Ability to Detect AI-Generated Deceptions
Generative artificial intelligence (GAI) is reshaping cybersecurity by enabling increasingly sophisticated attacks such as AI-generated phishing emails and deepfake videos. While prior research emphasizes technological defenses and adversarial risks, less is known about how routine AI use influences individuals’ ability to detect AI-generated deception. Drawing on cognitive offloading theory and media richness theory, this study examines how AI workflow assimilation, distinguishing between automation and augmentation, affects deception detection capability. We propose that automation, characterized by substitutive AI use, reduces cognitive vigilance and impairs detection accuracy, whereas augmentation, involving complementary human–AI collaboration, enhances analytical engagement and improves detection performance. Further, we argue that these effects are moderated by media richness, with stronger impacts under high-richness threats such as deepfake videos compared to text-based phishing. This study will advance a socio-cognitive perspective on AI-enabled cybersecurity vulnerabilities and contribute to understanding the human consequences of generative AI adoption.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIG SEC