Abstract
FGPE AuthorKit is an educational software belonging to the Framework for Gamified Programming Education (FGPE) ecosystem aimed at supporting course authors in developing gamified programming exercises. Recently, the tool has been enriched with GAMAI, a functional extension leveraging Large Language Models to enable the teachers to automatically generate gamified programming exercises based on a textual story input. In this paper, we report the results of the UMUX-based evaluation of perceived usability of the AI-enriched tool in context of the earlier evaluation of the same tool yet lacking the AI extension. The results reported here are quite unexpected as the UMUX score attained by the AI-supported version (79.5) is only less than six points higher than the original version’s (73.6), despite the way users interact with the software has been deeply transformed. Such an outcome brings implications for survey-based usability assessment of AI-enhanced software systems – not only educational.
Paper Type
Short Paper
DOI
10.62036/ISD.2026.158
The Effect of Enhancing Software with AI on Its Perceived Usability: The Case of FGPE AuthorKit
FGPE AuthorKit is an educational software belonging to the Framework for Gamified Programming Education (FGPE) ecosystem aimed at supporting course authors in developing gamified programming exercises. Recently, the tool has been enriched with GAMAI, a functional extension leveraging Large Language Models to enable the teachers to automatically generate gamified programming exercises based on a textual story input. In this paper, we report the results of the UMUX-based evaluation of perceived usability of the AI-enriched tool in context of the earlier evaluation of the same tool yet lacking the AI extension. The results reported here are quite unexpected as the UMUX score attained by the AI-supported version (79.5) is only less than six points higher than the original version’s (73.6), despite the way users interact with the software has been deeply transformed. Such an outcome brings implications for survey-based usability assessment of AI-enhanced software systems – not only educational.
Recommended Citation
Montella, R., De Vita, C.G., Mellone, G., Perrotta, S., Coppola, C., Queirós, R., Paiva, J.C. & Swacha, J.(2026). The Effect of Enhancing Software with AI on Its Perceived Usability: The Case of FGPE AuthorKit. In M. Valenta, B. Mannová, R. Pergl, A. Przybylek, M. Lang, H. Linger, C. Schneider, N. Iivari, & E. Insfran (Eds.), Making ISD Sustainable: Reloaded with AI and Automation (ISD2026 Proceedings). Prague, Czech Republic: Czech Technical University in Prague. ISBN: 978-80-01-07585-2. https://doi.org/10.62036/ISD.2026.158