•  
  •  
 
Communications of the Association for Information Systems

Author ORCID Identifier

Hamed Zolbanin: 0000-0001-5783-1495

Roopa Raman: 0000-0003-1740-6229

Behrooz Davazdahemami: 0000-0003-2885-6014

Jyotishka Ray: 0000-0001-5060-7650

Abstract

This exploratory study illuminates the “black box” of student reasoning by examining how Information Systems students engaged with Generative AI tools (ChatGPT) during a complex SQL exam. Using qualitative thematic analysis of 33 interaction transcripts, we identified preliminary behavioral patterns that differentiate successful from unsuccessful problem-solving. In this sense, these interactions resemble a digital think-aloud, in which AI transcripts provide insights into the internal thought processes associated with problem-solving. Analysis suggests a spectrum of interactions from “Strategic AI Engagement” (the “pilot”) to “Procedural AI Dependency” (the “passenger”). Successful engagements appeared to involve iterative refinement, systematic validation, and adaptive strategy shifts. Conversely, unsuccessful interactions tended to reflect a conceptual bypass characterized by passive query reception, rigid approaches, and an abdication of technical reasoning to the AI. The findings suggest that as technical proficiency shifts from procedural code generation toward evaluative judgment, educators should design AI-assisted assignments that prioritize auditing, verification, and strategic questioning. More broadly, the study suggests that GenAI can support but not replace the foundational competence and reflective judgment essential for effective problem-solving in data-driven professional environments.

Download

Share

COinS
 

When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.