Investigating the Effect of Sample Size and Respondent Characteristics on Usability Measurement: The Case of ChatGPT

Presenter Information

Jakub Swacha, University of Szczecin, PolandFollow
Łukasz Radliński, West Pomeranian University of Technology in Szczecin, PolandFollow
Karolina Muszyńska, University of Szczecin, PolandFollow
Susanne Marx, Technical University of Applied Sciences Wildau, GermanyFollow
Ricardo Queirós, Polytechnic of Porto, PortugalFollow

Abstract

Standardized usability questionnaires are a fast and relatively effortless way of assessing usability of software products. Despite their long use, so far, little attention has been paid to the effect of sample size and the level of respondents’ acquaintance with the evaluated software on the measurement. This paper addresses this gap and uses SUS and mTAM measurements of ChatGPT to illustrate how the deviation from the mean usability score decreases with increasing sample size, and to confirm the significant effect of usage frequency and knowledge of the evaluated software and its alternatives on the measurement results. It also exposes no demographic bias due to participation of respondents of different gender, country of stay, and academic major.

Recommended Citation

Swacha, J., Radliński, Ł., Muszyńska, K., Marx, S. & Queirós, R. (2025). Investigating the Effect of Sample Size and Respondent Characteristics on Usability Measurement: The Case of ChatGPTIn I. Luković, S. Bjeladinović, B. Delibašić, D. Barać, N. Iivari, E. Insfran, M. Lang, H. Linger, & C. Schneider (Eds.), Empowering the Interdisciplinary Role of ISD in Addressing Contemporary Issues in Digital Transformation: How Data Science and Generative AI Contributes to ISD (ISD2025 Proceedings). Belgrade, Serbia: University of Gdańsk, Department of Business Informatics & University of Belgrade, Faculty of Organizational Sciences. ISBN: 978-83-972632-1-5. https://doi.org/10.62036/ISD.2025.107

Paper Type

Full Paper

DOI

10.62036/ISD.2025.107