Paper Type
ERF
Abstract
This study addresses a central bottleneck in competitive dynamics research: the inability to measure competitive moves at scale without sacrificing construct validity. Prior work relies on manual content analysis of news articles, which provides rigor but limits the ability to examine long-term and large-scale competitive behavior. We propose a structured approach that treats Large Language Models (LLMs) as calibrated measurement instruments rather than automation tools. Grounded in competitive dynamics theory, the study constructs a gold standard of competitive moves from the Nation’s Restaurant News (NRN)s archive and uses it to calibrate an LLM classifier aligned with a shared codebook and evidence-grounding requirements. The resulting pipeline enables large-scale, traceable identification and classification of competitive moves, allowing the study of temporal patterns, firm-level heterogeneity, and sequences of competitive actions that are difficult to observe using manual approaches. This research contributes (1) a reviewed, inter-rater reliability gold standard of competitive moves in the U.S. restaurant industry; (2) a scalable LLM-based measurement approach calibrated to that benchmark for corpus-level classification; and (3) a traceable pipeline that links each classified competitive move to the original article snapshot and supporting text, making the results auditable and suitable for longitudinal analysis across large archives.
Paper Number
1600
Recommended Citation
Carassato Fernandes Brites, Lucas; Piccoli, Gabriele; Rodriguez, Joaquin; and Kim, Sun Tae, "Scale or Validity? Measuring Competitive Moves with Validated LLM Coding" (2026). AMCIS 2026 Proceedings. 3.
https://aisel.aisnet.org/amcis2026/scuidt/scuidt/3
Scale or Validity? Measuring Competitive Moves with Validated LLM Coding
This study addresses a central bottleneck in competitive dynamics research: the inability to measure competitive moves at scale without sacrificing construct validity. Prior work relies on manual content analysis of news articles, which provides rigor but limits the ability to examine long-term and large-scale competitive behavior. We propose a structured approach that treats Large Language Models (LLMs) as calibrated measurement instruments rather than automation tools. Grounded in competitive dynamics theory, the study constructs a gold standard of competitive moves from the Nation’s Restaurant News (NRN)s archive and uses it to calibrate an LLM classifier aligned with a shared codebook and evidence-grounding requirements. The resulting pipeline enables large-scale, traceable identification and classification of competitive moves, allowing the study of temporal patterns, firm-level heterogeneity, and sequences of competitive actions that are difficult to observe using manual approaches. This research contributes (1) a reviewed, inter-rater reliability gold standard of competitive moves in the U.S. restaurant industry; (2) a scalable LLM-based measurement approach calibrated to that benchmark for corpus-level classification; and (3) a traceable pipeline that links each classified competitive move to the original article snapshot and supporting text, making the results auditable and suitable for longitudinal analysis across large archives.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SCUIDT