Paper Type
ERF
Description
ChatGPT, an AI agent capable of answering questions from virtually all domains of knowledge, has created renewed concerns about the adverse effects of AI on students’ learning through cheating. We conducted an experiment involving a typical college course assignment to detect and compare genuine student responses to responses generated by ChatGPT. Using a text classification scheme, we showed that student responses are fairly accurately discriminable from AI’s, not only when AI uses its general knowledge to answer questions, but also when it is prompted to use the same material used by students. In addition, we identify authorship styles that set students’ and AI’s work apart. Specifically, we find AI’s language to be more formal and frozen. Together, results provide a machine learning model for detecting text generated by AI, as well as some stylistic cues that can help humans detect such text.
Paper Number
1918
Recommended Citation
Safi, Roozmehr and Jalali Naini, Azadeh, "The Work of Students and ChatGPT Compared: Using Machine Learning to Detect and Characterize AI-Generated Text" (2023). AMCIS 2023 Proceedings. 17.
https://aisel.aisnet.org/amcis2023/sig_aiaa/sig_aiaa/17
The Work of Students and ChatGPT Compared: Using Machine Learning to Detect and Characterize AI-Generated Text
ChatGPT, an AI agent capable of answering questions from virtually all domains of knowledge, has created renewed concerns about the adverse effects of AI on students’ learning through cheating. We conducted an experiment involving a typical college course assignment to detect and compare genuine student responses to responses generated by ChatGPT. Using a text classification scheme, we showed that student responses are fairly accurately discriminable from AI’s, not only when AI uses its general knowledge to answer questions, but also when it is prompted to use the same material used by students. In addition, we identify authorship styles that set students’ and AI’s work apart. Specifically, we find AI’s language to be more formal and frozen. Together, results provide a machine learning model for detecting text generated by AI, as well as some stylistic cues that can help humans detect such text.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIG AIAA