•  
  •  
 
Communications of the Association for Information Systems

Author ORCID Identifier

Daniel E. O’Leary: https://orcid.org/0000-0002-5240-9516

Veda C. Storey: https://orcid.org/0000-0002-8735-1553

Aaron M. French: https://orcid.org/0000-0002-7634-9925

Joseph R. Buckman: https://orcid.org/0000-0001-6883-6072

Cecil Chua: https://orcid.org/0000-0001-9384-1535

Andrew William Green: https://orcid.org/0000-0002-4727-715X

Grace Gu: https://orcid.org/0000-0002-7996-5483

Fred Niederman: https://orcid.org/0000-0003-3246-4769

Linda Wallace: https://orcid.org/0000-0002-6705-3604

Abstract

Large language models such as ChatGPT provide efficient access to a wealth of information. However, there are significant questions regarding the depth and quality of knowledge in any one domain. This paper focuses specifically on the information systems (IS) field and assesses ChatGPT’s knowledge. To analyze the extent and quality of information systems knowledge derived from queries to ChatGPT, we used over 3,000 queries from a broad range of exam and quiz questions. These queries were obtained from university courses and professional information system certification exams. The query topics are based on a framework for information systems education, with queries gathered from multiple faculty members at different universities. The results of querying ChatGPT with these questions find that ChatGPT can answer 65% to 85% of information systems queries successfully, across each of the areas of the framework. ChatGPT performed well on essay, true/false and multiple-choice questions, with no statistical difference between its success rate on multiple-choice and true/false questions. We conclude that ChatGPT tends to perform approximately at the level of an average student, but may not perform at a level sufficient to pass certain professional exams. ChatGPT’s knowledge is very broad, covering virtually all areas of the information systems knowledge we identified, but not at an expert level.

Share

COinS
 

When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.