Start Date
11-12-2016 12:00 AM
Description
A critical bottleneck in crowdsourced innovation challenges is the process of reviewing and selecting the best submissions. This bottleneck is especially problematic in settings where submissions are complex intellectual artifacts whose evaluation requires expertise. To help reduce the review load from experts, we offer a computational approach that relies on analyzing sociolinguistic and other characteristics of submission text, as well as activities of the crowd and the submission authors, and scores the submissions. We developed and tested models based on data from contests done in a large citizen-science platform - the Climate CoLab - and find that they are able to accurately predict expert decisions about the submissions, and can lead to substantial reduction of review labor, and acceleration of the review process.
Recommended Citation
Nagar, Yiftach; de Boer, Patrick; and Bicharra Garcia, Ana Cristina, "Accelerating the Review of Complex Intellectual Artifacts in Crowdsourced Innovation Challenges" (2016). ICIS 2016 Proceedings. 16.
https://aisel.aisnet.org/icis2016/Crowdsourcing/Presentations/16
Accelerating the Review of Complex Intellectual Artifacts in Crowdsourced Innovation Challenges
A critical bottleneck in crowdsourced innovation challenges is the process of reviewing and selecting the best submissions. This bottleneck is especially problematic in settings where submissions are complex intellectual artifacts whose evaluation requires expertise. To help reduce the review load from experts, we offer a computational approach that relies on analyzing sociolinguistic and other characteristics of submission text, as well as activities of the crowd and the submission authors, and scores the submissions. We developed and tested models based on data from contests done in a large citizen-science platform - the Climate CoLab - and find that they are able to accurately predict expert decisions about the submissions, and can lead to substantial reduction of review labor, and acceleration of the review process.