Paper Number
ICIS2025-2704
Paper Type
Complete
Abstract
Ensuring the quality of self-reported survey data remains a persistent challenge in information systems (IS) research, particularly when respondents provide non-thoughtful answers that compromise data validity. Common detection methods--such as attention checks and overall survey duration--often fall short, as they are easily gamed or fail to reflect genuine cognitive engagement. To address these limitations, this study draws on the Metamemory framework to investigate whether fine-grained behavioral metrics can unobtrusively indicate cognitive engagement in an online survey and help identify non-thoughtful responses. In a controlled experiment, we found that respondents who thoughtfully engaged with the survey spent more time on each question, clicked more frequently, and exhibited greater cursor movement deviation. These metrics also show promise for use in predictive models to identify non-thoughtful responses. This research contributes to the IS field by linking metacognitive theory to online interaction behavior and advancing scalable methods for enhancing survey data integrity.
Recommended Citation
Jenkins, Jeffrey L.; Shuldberg, Coleman A.; Valacich, Joseph S.; Wilson, David; Weisgarber, Paul A.; and Kim, David, "From Clicks to Cognition: Detecting Thoughtful Engagement in Digital Surveys Through Behavioral Metrics" (2025). ICIS 2025 Proceedings. 34.
https://aisel.aisnet.org/icis2025/user_behav/user_behav/34
From Clicks to Cognition: Detecting Thoughtful Engagement in Digital Surveys Through Behavioral Metrics
Ensuring the quality of self-reported survey data remains a persistent challenge in information systems (IS) research, particularly when respondents provide non-thoughtful answers that compromise data validity. Common detection methods--such as attention checks and overall survey duration--often fall short, as they are easily gamed or fail to reflect genuine cognitive engagement. To address these limitations, this study draws on the Metamemory framework to investigate whether fine-grained behavioral metrics can unobtrusively indicate cognitive engagement in an online survey and help identify non-thoughtful responses. In a controlled experiment, we found that respondents who thoughtfully engaged with the survey spent more time on each question, clicked more frequently, and exhibited greater cursor movement deviation. These metrics also show promise for use in predictive models to identify non-thoughtful responses. This research contributes to the IS field by linking metacognitive theory to online interaction behavior and advancing scalable methods for enhancing survey data integrity.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
16-UserBehavior