Abstract
Large language models (LLMs) are increasingly used as on-demand assistants for software tasks, yet their educational value under everyday, unguided use remains uncertain. This study tests whether unrestricted LLM access improves beginners’ code comprehension. In a controlled classroom experiment, undergraduate novices practiced comprehension tasks in Python, SQL, and HTML/CSS either with unrestricted access to a public LLM or without AI support. Matched pre- and post-tests measured learning gains, with LLM access disabled during testing. Both groups improved modestly; however, mean gains did not differ significantly between conditions across languages, and effect sizes were small. The results indicate that LLM availability alone is insufficient to measurably improve novice code comprehension without instructional scaffolding, and they motivate governance and training approaches for sustainable AI-assisted information systems development.
Paper Type
Poster
DOI
10.62036/ISD.2026.179
The Unnoticeable Effect of Using LLMs in Computer Science Education
Large language models (LLMs) are increasingly used as on-demand assistants for software tasks, yet their educational value under everyday, unguided use remains uncertain. This study tests whether unrestricted LLM access improves beginners’ code comprehension. In a controlled classroom experiment, undergraduate novices practiced comprehension tasks in Python, SQL, and HTML/CSS either with unrestricted access to a public LLM or without AI support. Matched pre- and post-tests measured learning gains, with LLM access disabled during testing. Both groups improved modestly; however, mean gains did not differ significantly between conditions across languages, and effect sizes were small. The results indicate that LLM availability alone is insufficient to measurably improve novice code comprehension without instructional scaffolding, and they motivate governance and training approaches for sustainable AI-assisted information systems development.
Recommended Citation
Naprawski, T., Pilarczyk-Naprawski, A., Swacha, J. & Kowalik, J.(2026). The Unnoticeable Effect of Using LLMs in Computer Science Education. In M. Valenta, B. Mannová, R. Pergl, A. Przybylek, M. Lang, H. Linger, C. Schneider, N. Iivari, & E. Insfran (Eds.), Making ISD Sustainable: Reloaded with AI and Automation (ISD2026 Proceedings). Prague, Czech Republic: Czech Technical University in Prague. ISBN: 978-80-01-07585-2. https://doi.org/10.62036/ISD.2026.179