Sharing Economy, Platforms and Crowds
Loading...
Paper Number
1762
Paper Type
short
Description
Crowdsourced idea evaluation is often seen as a viable, less expensive alternative to traditional idea selection mechanisms such as expert evaluation. However, previous research found several limitations on the reliability of crowdsourced decision making, especially when simple majority-based aggregation mechanisms are used. In response, researchers suggest new, advanced aggregation mechanisms that rely on additional information (e.g. the confidence of a judge) to overcome these limitations. Therefore, this study investigates whether advanced aggregation methods based on the judge’s confidence can be used to increase the reliability of crowdsourced idea evaluation. To answer this question, we post an idea evaluation task to a crowdworking platform, aggregate the judgements with different mechanisms and compare the predictions from each mechanism with results from expert evaluation. Our findings suggest that confidence-based mechanisms reduce the risk of discarding high quality ideas, at the cost of a slight increase in the misclassification rate on low quality ideas.
Recommended Citation
Laux, Florian and Görzen, Thomas, "Trust Me, I’m Confident – Are Confident Members of the Crowd Better at Evaluating Business Model Ideas?" (2021). ICIS 2021 Proceedings. 10.
https://aisel.aisnet.org/icis2021/sharing_econ/sharing_econ/10
Trust Me, I’m Confident – Are Confident Members of the Crowd Better at Evaluating Business Model Ideas?
Crowdsourced idea evaluation is often seen as a viable, less expensive alternative to traditional idea selection mechanisms such as expert evaluation. However, previous research found several limitations on the reliability of crowdsourced decision making, especially when simple majority-based aggregation mechanisms are used. In response, researchers suggest new, advanced aggregation mechanisms that rely on additional information (e.g. the confidence of a judge) to overcome these limitations. Therefore, this study investigates whether advanced aggregation methods based on the judge’s confidence can be used to increase the reliability of crowdsourced idea evaluation. To answer this question, we post an idea evaluation task to a crowdworking platform, aggregate the judgements with different mechanisms and compare the predictions from each mechanism with results from expert evaluation. Our findings suggest that confidence-based mechanisms reduce the risk of discarding high quality ideas, at the cost of a slight increase in the misclassification rate on low quality ideas.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
09-Crowds