•  
  •  
 
Communications of the Association for Information Systems

Author ORCID Identifier

Alan Dennis: https://orcid.org/0000-0002-6439-6134

Warren Rosengren: https://orcid.org/0009-0006-6483-3759

Joseph Steed: https://orcid.org/0000-0002-0365-1453

Tucker Todd: https://orcid.org/0009-0005-5411-2774

Abstract

Academic research often produces unstructured data, which has no ground truth, that must be coded using structured rubrics for analysis. Valid coding requires that independent coders applying the same instructions produce similar results, as high agreement indicates reliability. Traditionally, this work has relied on human coders; however, human coding is slow, costly, and prone to inconsistency. This paper proposes a five-step framework that integrates generative artificial intelligence (AI) with human coders and inter-rater reliability assessments to deliver faster, transparent, and replicable coding. The process: (1) develops a rigorous frame-of-reference rubric; (2) assesses human reliability on a subset of data; (3) evaluates AI reliability against human coders on that subset; (4) uses an ensemble of large language models (LLMs) to code the full dataset and assess agreement across models; and (5) applies a Delphi-style process allowing LLMs to revise scores after seeing one another’s outputs. We illustrate the framework with two studies. The first used five LLMs to score more than 2,500 participant-generated ideas on novelty, workability, and relevance, achieving sufficient reliability levels for analysis, comparable to human coding. The second applied the method to a different dataset using a different set of three LLMs and again achieved acceptable reliability for analysis. We offer guidelines and informed suggestions for prompt design, tool selection, bias checks, and opportunities for large-scale qualitative research.

Download

Share

COinS
 

When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.