Start Date
14-12-2012 12:00 AM
Description
Arranging survey items to group measures of the same construct together has several benefits, including ease of administration and enhanced statistical reliability and validity of constructs. Yet some IS researchers claim this practice contributes to common methods bias and camouflages “true” measures of reliability. Our study takes a new approach to this issue by using a range of IS research measures in an online survey context to contrast grouped-item survey administration with a design in which the ordering of item administration is programmatically re-randomized for each individual subject. We find significant differences in construct reliability between grouped-item and individually randomized treatments as well as strong temporal effects and widespread anomalies related to item-ordering in the grouped-item treatment. Our results suggest the purported benefits of grouped-item surveys are outweighed by hazards these create to the integrity of research findings, and we caution IS researchers against their continued use.
Recommended Citation
Wilson, Vance and Lankton, Nancy, "Some Unfortunate Consequences of Non-Randomized, Grouped-Item Survey Administration in IS Research" (2012). ICIS 2012 Proceedings. 4.
https://aisel.aisnet.org/icis2012/proceedings/ResearchMethods/4
Some Unfortunate Consequences of Non-Randomized, Grouped-Item Survey Administration in IS Research
Arranging survey items to group measures of the same construct together has several benefits, including ease of administration and enhanced statistical reliability and validity of constructs. Yet some IS researchers claim this practice contributes to common methods bias and camouflages “true” measures of reliability. Our study takes a new approach to this issue by using a range of IS research measures in an online survey context to contrast grouped-item survey administration with a design in which the ordering of item administration is programmatically re-randomized for each individual subject. We find significant differences in construct reliability between grouped-item and individually randomized treatments as well as strong temporal effects and widespread anomalies related to item-ordering in the grouped-item treatment. Our results suggest the purported benefits of grouped-item surveys are outweighed by hazards these create to the integrity of research findings, and we caution IS researchers against their continued use.