Studying the GenAI’s Impact on Learning and Critical Thinking: A Multi-Site Field Experiment

Presenter Information

Isabel Ramos, University of MinhoFollow
Anabela Mesquita, Polytechnic of PortoFollow
António Luís Lopes, Instituto Universitário de Lisboa (ISCTE-IUL)Follow
Fernando Moreira, Universidade PortucalenseFollow
Victor Barros, University of MinhoFollow
Rui Dinis Sousa, University of MinhoFollow
Gonçalo Regadas, Universidade do MinhoFollow
Luis Gomes, Universidade de ÉvoraFollow
Patricia Leite, Escola Superior de Tecnologia do Instituto Politécnico do Cavado e do AveFollow

Paper Number

ICIS2025-1600

Paper Type

Short

Abstract

Generative AI (GenAI) tools are transforming higher education, but robust evidence of their pedagogical value is limited. This multi-site field experiment, based on Situated Cognition Theory, explores the impact of GenAI on learning and critical thinking at ten universities and polytechnics. Around 270 students were randomly assigned to an experimental group using licensed ChatGPT for a supervised task, or a control group using traditional resources. Tasks, aligned with local learning outcomes, were evaluated using a rubric assessing critical thinking, accuracy, coherence, and originality. Classroom observations and a post-task questionnaire further enriched the analysis. Preliminary results indicate that ChatGPT enhanced content comprehension and motivation, but many students struggled with evaluating AI reliability and creating effective prompts.

Comments

24-Learning

Recommended Citation

Ramos, Isabel; Mesquita, Anabela; Lopes, António Luís; Moreira, Fernando; Barros, Victor; Sousa, Rui Dinis; Regadas, Gonçalo; Gomes, Luis; and Leite, Patricia, "Studying the GenAI’s Impact on Learning and Critical Thinking: A Multi-Site Field Experiment" (2025). ICIS 2025 Proceedings. 5.
https://aisel.aisnet.org/icis2025/learn_curricula/learn_curricula/5