Paper Type
Complete
Abstract
ChatGPT, a chat interface to OpenAI’s Large Language Model (LLM), amazed users after its release in 2022 with its functionality. However, LLMs are prone to “hallucinations”, which sparked lively debate about LLM reasoning capabilities, which is also the focus of this study. We chose logical inferencing, which is the cognitive process of drawing conclusions from logical assertions. We utilised computational ontologies, which, because of the logical frameworks they employ, provide a comparable reasoning baseline. We developed an ontology with an inconsistent concept, and we provided the same assertions as a zero-shot prompt for several LLMs. Even though all the LLMs provided quite an extensive explanation for their answers, all but two of them failed the basic logical inferencing experiment. The results indicate that reasoning remains a challenge for LLMs but that some of the latest LLMs have progressed significantly in their logical reasoning capabilities.
Paper Number
1035
Recommended Citation
Gerber, Aurona and Eybers, Sunet, "Can LLMs Reason with Logic?" (2025). AMCIS 2025 Proceedings. 34.
https://aisel.aisnet.org/amcis2025/intelfuture/intelfuture/34
Can LLMs Reason with Logic?
ChatGPT, a chat interface to OpenAI’s Large Language Model (LLM), amazed users after its release in 2022 with its functionality. However, LLMs are prone to “hallucinations”, which sparked lively debate about LLM reasoning capabilities, which is also the focus of this study. We chose logical inferencing, which is the cognitive process of drawing conclusions from logical assertions. We utilised computational ontologies, which, because of the logical frameworks they employ, provide a comparable reasoning baseline. We developed an ontology with an inconsistent concept, and we provided the same assertions as a zero-shot prompt for several LLMs. Even though all the LLMs provided quite an extensive explanation for their answers, all but two of them failed the basic logical inferencing experiment. The results indicate that reasoning remains a challenge for LLMs but that some of the latest LLMs have progressed significantly in their logical reasoning capabilities.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
IntelFuture