Management Information Systems Quarterly
Abstract
Online labor markets and the humans that power them serve a critical role in the advancement of artificial intelligence and supervised machine learning via the creation of useful training datasets. The use of human effort in online labor markets is not enough, however, as a key factor is understanding the possible interventions that market operators can leverage to incentivize human effort among their labor force. We propose that platforms could implement mechanisms such as rewards or punishments at individual or group levels to incentivize real-effort and output. We apply our interventions using a collaborative image tagging experiment—a folksonomy—and the results provide interesting insights and nonobvious consequences. On average, interventions applied at the group level outperformed interventions applied at the individual level. Punishing the group provided the most controversial incentive strategy and provided a nonobvious significant improvement in effort. Rewarding or sanctioning an individual had similar effects on average, with both treatments leading to significant increases in effort post-intervention. In contrast to predictions, sanctioning appears to have significantly motivated those that were punished. Overall, the interventions applied in our real-effort collaborative image tagging experiment had a significant impact on behavior, which provides guidance for online labor market operators and the use of incentives in the creation of labeled machine learning training datasets.