Paper Number
ICIS2025-2529
Paper Type
Complete
Abstract
The rise of generative AI and large language models (LLMs) has sparked interest in applying generative agents to simulate users in conversational recommender system (CRS) evaluations. While CRS rely on user perception for success, traditional user studies are costly and time-intensive. Existing simulation approaches lack interaction depth and user-centric evaluation. This study addresses these gaps by leveraging LLM-based generative agents to conduct scalable, subjective CRS assessments. We present a comparative study where 200 generative agents and 50 human participants interact with a prototypical agentic CRS and evaluate their experience using a structured questionnaire. Results indicate that generative agents approximate human-like behavior and subjective assessments at the macro-level, despite granular precision limitations, offering an alternative to traditional user studies. Our findings advance research on CRS evaluation by demonstrating how agentic simulations can support human-aligned assessments of socio-technical systems and open new avenues for applying Generative AI in user-centered decision support.
Recommended Citation
Holldack, Florian; Banh, Leonardo; and Strobel, Gero, "Synthetic Conversations, Real Insights: Towards Realistic User Simulations with Generative Agents" (2025). ICIS 2025 Proceedings. 16.
https://aisel.aisnet.org/icis2025/isdesign/isdesign/16
Synthetic Conversations, Real Insights: Towards Realistic User Simulations with Generative Agents
The rise of generative AI and large language models (LLMs) has sparked interest in applying generative agents to simulate users in conversational recommender system (CRS) evaluations. While CRS rely on user perception for success, traditional user studies are costly and time-intensive. Existing simulation approaches lack interaction depth and user-centric evaluation. This study addresses these gaps by leveraging LLM-based generative agents to conduct scalable, subjective CRS assessments. We present a comparative study where 200 generative agents and 50 human participants interact with a prototypical agentic CRS and evaluate their experience using a structured questionnaire. Results indicate that generative agents approximate human-like behavior and subjective assessments at the macro-level, despite granular precision limitations, offering an alternative to traditional user studies. Our findings advance research on CRS evaluation by demonstrating how agentic simulations can support human-aligned assessments of socio-technical systems and open new avenues for applying Generative AI in user-centered decision support.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
13-DesignDevPM