Paper Number
ECIS2026-2633
Paper Type
CRP
Abstract
For labeling data, until today, Crowd Labeling is the mechanism of choice. In the practice of Crowd Labeling, a bunch of various instruments is used, which allow for controlling the crowd, respectively its work results. However, these instruments have in common that they can be executed only before or after a labelling campaign. This causes large delays and therefore not only needs a lot of additional time but also raises extra costs and -effort. In our design science research, we developed and evaluated a Deep Learning-based, algorithmic approach consisting of four different probabilistic-mathematical models. Our new approach enables an assessment of the crowd’s labeling work during an ongoing crowdwork campaign. Besides this benefit, our approach produces excellent rates of accuracy, when it comes to detecting erroneous annotation rates of the crowd.
Recommended Citation
Bretschneider, Ulrich; Hupe, Anna Lena; Herde, Marek; Oeste-Reiß, Sarah; and Sick, Bernhard, "The Good Into The Pot, The Bad Into The Crop: A Deep Learning Approach To Control Crowd Labeling" (2026). ECIS 2026 Proceedings. 8.
https://aisel.aisnet.org/ecis2026/algo_fow/algo_fow/8
The Good Into The Pot, The Bad Into The Crop: A Deep Learning Approach To Control Crowd Labeling
For labeling data, until today, Crowd Labeling is the mechanism of choice. In the practice of Crowd Labeling, a bunch of various instruments is used, which allow for controlling the crowd, respectively its work results. However, these instruments have in common that they can be executed only before or after a labelling campaign. This causes large delays and therefore not only needs a lot of additional time but also raises extra costs and -effort. In our design science research, we developed and evaluated a Deep Learning-based, algorithmic approach consisting of four different probabilistic-mathematical models. Our new approach enables an assessment of the crowd’s labeling work during an ongoing crowdwork campaign. Besides this benefit, our approach produces excellent rates of accuracy, when it comes to detecting erroneous annotation rates of the crowd.