Abstract

The growing adoption of online research methods and the emergence of generative AI (GenAI) pose critical challenges to ensuring data quality and authenticity in behavioral studies. Traditional attention checks and bot detection methods are increasingly insufficient in distinguishing genuine human responses from sophisticated AI-generated content. To address this issue, we introduce Screen Tracking of Research Engagement (STORE), an innovative methodology that employs continuous video-based monitoring of participant interactions within the research environment. By capturing detailed clickstreams, navigation paths, and interaction patterns, STORE not only verifies the human origin of data but also provides rich contextual insights into participants' cognitive processes and decision-making strategies. We demonstrate the effectiveness of STORE through a use case involving programmers' interactions with human- and AI-generated solutions, showcasing its ability to segment problem-solving phases, detect hallucinations, and track solution adoption. STORE significantly enhances transparency and data integrity while enabling comprehensive mixed-methods research that integrates quantitative metrics with qualitative observations. We discuss the methodological advantages, ethical considerations, and future research directions for refining and expanding the application of STORE across various domains. By bridging empirical rigor with theoretical depth, STORE advances our understanding of human-GenAI dynamics and contributes to the development of robust frameworks for studying socio-technical systems in the era of GenAI.

Download

Share

COinS