Paper Number
ICIS2025-1313
Paper Type
Complete
Abstract
Researchers increasingly use paid online panels to collect survey data. However, financial incentives may lead to inappropriate participant self-selection, compromising data quality and relevance. Despite these concerns, the fundamental question of how participants are screened into surveys has received surprisingly scant attention. This study examines the effect of alternative survey framing approaches on screen-out rates, attentiveness, and data quality. Two samples were collected via Prolific: one with participant opt-in based on explicit topic disclosure (n = 1,035), and the other using hidden framing with screening criteria concealed from respondents (n = 695). Analysis revealed a substantial difference in screen-out rates (explicit: 11.87%, hidden: 78.33%), but no significant difference in data quality based on reliability or convergent and discriminant validity. However, common method variance was unevenly distributed in the explicit framing sample and evenly distributed in the hidden framing sample. Based on these findings, we offer practical recommendations for survey researchers.
Recommended Citation
Jalo, Henri; Makkonen, Markus; Pirkkalainen, Henri; and Seppänen, Marko, "Impact of Explicit and Hidden Survey Framing on Paid Online Survey Panel Participant Screening Rates, Attentiveness, and Data Quality: An Empirical Examination" (2025). ICIS 2025 Proceedings. 4.
https://aisel.aisnet.org/icis2025/is_researchmethods/is_researchmethods/4
Impact of Explicit and Hidden Survey Framing on Paid Online Survey Panel Participant Screening Rates, Attentiveness, and Data Quality: An Empirical Examination
Researchers increasingly use paid online panels to collect survey data. However, financial incentives may lead to inappropriate participant self-selection, compromising data quality and relevance. Despite these concerns, the fundamental question of how participants are screened into surveys has received surprisingly scant attention. This study examines the effect of alternative survey framing approaches on screen-out rates, attentiveness, and data quality. Two samples were collected via Prolific: one with participant opt-in based on explicit topic disclosure (n = 1,035), and the other using hidden framing with screening criteria concealed from respondents (n = 695). Analysis revealed a substantial difference in screen-out rates (explicit: 11.87%, hidden: 78.33%), but no significant difference in data quality based on reliability or convergent and discriminant validity. However, common method variance was unevenly distributed in the explicit framing sample and evenly distributed in the hidden framing sample. Based on these findings, we offer practical recommendations for survey researchers.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
25-Research