Paper Number
ECIS2026-1203
Paper Type
CRP
Abstract
Large Language Models (LLMs) are transforming research practices in Information Systems (IS), offering new opportunities for qualitative data analysis. While recent studies show that LLM-assisted coding can achieve human-comparable accuracy, challenges such as hallucinations, inconsistency, and limited methodological guidance remain. To address these issues, we propose SCALE, a structured framework for scaling qualitative data coding. SCALE integrates human implicit understanding into LLM-assisted coding and scales it. Additionally, it establishes systematic validation across the entire coding process. SCALE introduces steps for rigorous project design, gold-standard creation, iterative prompt refinement, and multi-stage validation. Through this approach, SCALE balances scalability with reliability, enabling researchers to conduct large-scale qualitative analyses efficiently. Our evaluation shows that structured processes can align LLM outputs with human coding while reducing manual effort, contributing to a more robust methodological foundation for artificial intelligence (AI)-assisted qualitative research in IS.
Recommended Citation
Rogge, Stefan; Weiß, Marco Maximilian; Ingendahl, Franziska; Undorf, Monika; and Jussupow, Ekaterina, "Scale: Scaling Qualitative Data Coding With LLMs" (2026). ECIS 2026 Proceedings. 4.
https://aisel.aisnet.org/ecis2026/gen_track/gen_track/4
Scale: Scaling Qualitative Data Coding With LLMs
Large Language Models (LLMs) are transforming research practices in Information Systems (IS), offering new opportunities for qualitative data analysis. While recent studies show that LLM-assisted coding can achieve human-comparable accuracy, challenges such as hallucinations, inconsistency, and limited methodological guidance remain. To address these issues, we propose SCALE, a structured framework for scaling qualitative data coding. SCALE integrates human implicit understanding into LLM-assisted coding and scales it. Additionally, it establishes systematic validation across the entire coding process. SCALE introduces steps for rigorous project design, gold-standard creation, iterative prompt refinement, and multi-stage validation. Through this approach, SCALE balances scalability with reliability, enabling researchers to conduct large-scale qualitative analyses efficiently. Our evaluation shows that structured processes can align LLM outputs with human coding while reducing manual effort, contributing to a more robust methodological foundation for artificial intelligence (AI)-assisted qualitative research in IS.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.