Abstract
Requirements elicitation is a resource-intensive task that artificial intelligence (AI) can potentially streamline. This paper aims to compare large language model (LLM)-generated aggregated response distributions with requirements from human participants for an app designed to enhance creativity during training sessions. The exploratory research is based on: a questionnaire administered to human respondents, results from the same survey across LLMs - GPT, Gemini, Grok, Mistral, Claude; and a comparative analysis of results. The results show that LLMs can mimic preferences for minimalistic and intuitive interfaces. However, for more detailed aspects, differences emerge between human responses and LLM-generated aggregated response distributions. This study contributes to practice by demonstrating that LLMs can support the identification of general user preferences, but should not replace human users due to systematic biases.
Paper Type
Poster
DOI
10.62036/ISD.2026.28
Human Users vs. LLMs: An Empirical Study of Requirements Elicitation for Creativity Enhancement App
Requirements elicitation is a resource-intensive task that artificial intelligence (AI) can potentially streamline. This paper aims to compare large language model (LLM)-generated aggregated response distributions with requirements from human participants for an app designed to enhance creativity during training sessions. The exploratory research is based on: a questionnaire administered to human respondents, results from the same survey across LLMs - GPT, Gemini, Grok, Mistral, Claude; and a comparative analysis of results. The results show that LLMs can mimic preferences for minimalistic and intuitive interfaces. However, for more detailed aspects, differences emerge between human responses and LLM-generated aggregated response distributions. This study contributes to practice by demonstrating that LLMs can support the identification of general user preferences, but should not replace human users due to systematic biases.
Recommended Citation
Łuczak, K., Domagała, U., Renik, K. & Nowacka, A.(2026). Human Users vs. LLMs: An Empirical Study of Requirements Elicitation for Creativity Enhancement App. In M. Valenta, B. Mannová, R. Pergl, A. Przybylek, M. Lang, H. Linger, C. Schneider, N. Iivari, & E. Insfran (Eds.), Making ISD Sustainable: Reloaded with AI and Automation (ISD2026 Proceedings). Prague, Czech Republic: Czech Technical University in Prague. ISBN: 978-80-01-07585-2. https://doi.org/10.62036/ISD.2026.28