Author ORCID Identifier
Daniel E. O’Leary: https://orcid.org/0000-0002-5240-9516
Veda C. Storey: https://orcid.org/0000-0002-8735-1553
Aaron M. French: https://orcid.org/0000-0002-7634-9925
Joseph R. Buckman: https://orcid.org/0000-0001-6883-6072
Cecil Chua: https://orcid.org/0000-0001-9384-1535
Andrew William Green: https://orcid.org/0000-0002-4727-715X
Grace Gu: https://orcid.org/0000-0002-7996-5483
Fred Niederman: https://orcid.org/0000-0003-3246-4769
Linda Wallace: https://orcid.org/0000-0002-6705-3604
Abstract
Large language models such as ChatGPT provide efficient access to a wealth of information. However, there are significant questions regarding the depth and quality of knowledge in any one domain. This paper focuses specifically on the information systems (IS) field and assesses ChatGPT’s knowledge. To analyze the extent and quality of information systems knowledge derived from queries to ChatGPT, we used over 3,000 queries from a broad range of exam and quiz questions. These queries were obtained from university courses and professional information system certification exams. The query topics are based on a framework for information systems education, with queries gathered from multiple faculty members at different universities. The results of querying ChatGPT with these questions find that ChatGPT can answer 65% to 85% of information systems queries successfully, across each of the areas of the framework. ChatGPT performed well on essay, true/false and multiple-choice questions, with no statistical difference between its success rate on multiple-choice and true/false questions. We conclude that ChatGPT tends to perform approximately at the level of an average student, but may not perform at a level sufficient to pass certain professional exams. ChatGPT’s knowledge is very broad, covering virtually all areas of the information systems knowledge we identified, but not at an expert level.
Recommended Citation
O’Leary, D. E., Storey, V. C., French, A. M., Buckman, J. R., Chua, C., Green, A. W., Gu, G., Niederman, F., Pereira, F., Templeton, G., & Wallace, L. (In press). What Does ChatGPT Know About Information Systems?. Communications of the Association for Information Systems, 56, pp-pp. Retrieved from https://aisel.aisnet.org/cais/vol56/iss1/19
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
