Paper Number
ICIS2025-1070
Paper Type
Short
Abstract
Information Quality (IQ) has been traditionally understood with reference to accuracy, completeness, or consistency—of outputs of deterministic computations or of information retrieved from organizational databases. By contrast, the outputs of large language models (LLMs) are generated probabilistically, which makes it difficult to account for their quality within these traditional dimensions. To uncover how IQ is constituted in LLM outputs, we conducted interviews with 70 LLM trainers working for nine companies specializing in improving the quality of LLM outputs, supported by the analysis of 372 pages of secondary documentation. We found that IQ in LLM outputs is produced in a dynamic sociotechnical assemblage where both human and nonhuman entities perform practices aimed at improving IQ along dimensions that are enacted in practice. Our study is expected to extend IQ literature by theorizing IQ in generative AI applications, specifically LLMs, as well as providing practical insights on IQ in generative AI.
Recommended Citation
Stelmaszak, Marta; Xue, Yue; Tarafdar, Monideepa; and Aaltonen, Aleksi, "“Think About an LLM Like a Toddler”: Information Quality in Large Language Models’ Outputs" (2025). ICIS 2025 Proceedings. 3.
https://aisel.aisnet.org/icis2025/gen_ai/gen_ai/3
“Think About an LLM Like a Toddler”: Information Quality in Large Language Models’ Outputs
Information Quality (IQ) has been traditionally understood with reference to accuracy, completeness, or consistency—of outputs of deterministic computations or of information retrieved from organizational databases. By contrast, the outputs of large language models (LLMs) are generated probabilistically, which makes it difficult to account for their quality within these traditional dimensions. To uncover how IQ is constituted in LLM outputs, we conducted interviews with 70 LLM trainers working for nine companies specializing in improving the quality of LLM outputs, supported by the analysis of 372 pages of secondary documentation. We found that IQ in LLM outputs is produced in a dynamic sociotechnical assemblage where both human and nonhuman entities perform practices aimed at improving IQ along dimensions that are enacted in practice. Our study is expected to extend IQ literature by theorizing IQ in generative AI applications, specifically LLMs, as well as providing practical insights on IQ in generative AI.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
12-GenAI