Paper Type
Complete
Abstract
In this study, we explore automating the analysis steps in the procedurally rigorous qualitative analysis approach popular in the field of IS: the Gioia method. Using DeepSeek’s R1 chain-of-thought model, custom-built pipelines on a high-performance computing infrastructure, we sought to replicate a peer-reviewed and published original analysis of 17 expert interview transcripts. To address model hallucinations, we used Levenshtein distance to check that provided quotes exist in the transcript. We found that while the constructed pipeline produced concepts similar to the ones in the original publication, there were challenges in maintaining interpretive rigor (i.e., the system extracted 1st order concepts but lost meanings associated with them). This led the LLM to overgeneralize, ending up with 2nd order themes and aggregate dimensions that were inaccurate and borderline non-informative. We elaborate on nine unique challenges we encountered and provide directions for future research.
Paper Number
2130
Recommended Citation
Laato, Joonatan; Mäntymäki, Matti; Kordyaka, Bastian; and Laato, Samuli, "Automating Qualitative Data Analysis with Chain-of-Thought Reasoning Models: A Study with the Gioia Method" (2025). AMCIS 2025 Proceedings. 1.
https://aisel.aisnet.org/amcis2025/data_science/sig_dsa/1
Automating Qualitative Data Analysis with Chain-of-Thought Reasoning Models: A Study with the Gioia Method
In this study, we explore automating the analysis steps in the procedurally rigorous qualitative analysis approach popular in the field of IS: the Gioia method. Using DeepSeek’s R1 chain-of-thought model, custom-built pipelines on a high-performance computing infrastructure, we sought to replicate a peer-reviewed and published original analysis of 17 expert interview transcripts. To address model hallucinations, we used Levenshtein distance to check that provided quotes exist in the transcript. We found that while the constructed pipeline produced concepts similar to the ones in the original publication, there were challenges in maintaining interpretive rigor (i.e., the system extracted 1st order concepts but lost meanings associated with them). This led the LLM to overgeneralize, ending up with 2nd order themes and aggregate dimensions that were inaccurate and borderline non-informative. We elaborate on nine unique challenges we encountered and provide directions for future research.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIGDSA