Location
Hilton Waikoloa Village, Hawaii
Event Website
https://hicss.hawaii.edu/
Start Date
7-1-2025 12:00 AM
End Date
10-1-2025 12:00 AM
Description
The process of identifying relevant prior research articles is crucial for theoretical advancements, but often requires significant human effort. This study examines the feasibility of using large language models (LLMs) to support this task by extracting tested hypotheses, which consist of related constructs, moderators or mediators, path coefficients, and p-values, from empirical studies using structural equation modeling (SEM). We combine state-of-the-art LLMs with a variety of post-processing measures to improve the relation extraction quality. An extensive evaluation yields recall scores of up to 79.2% in construct entity extraction, 58.4% in construct-mediator/moderator-construct extraction, and 39.3% in extracting the full tested hypotheses. We provide a manually annotated dataset of 72 SEM articles and 749 construct relations to facilitate future research. Our findings offer critical insights and suggest promising directions for advancing the field of automated construct relation extraction from scholarly documents.
Recommended Citation
Scharfenberger, Jonas and Funk, Burkhardt, "Construct Relation Extraction from Scientific Papers: Is It Automatable Yet?" (2025). Hawaii International Conference on System Sciences 2025 (HICSS-58). 3.
https://aisel.aisnet.org/hicss-58/ks/ai_based_assistants/3
Construct Relation Extraction from Scientific Papers: Is It Automatable Yet?
Hilton Waikoloa Village, Hawaii
The process of identifying relevant prior research articles is crucial for theoretical advancements, but often requires significant human effort. This study examines the feasibility of using large language models (LLMs) to support this task by extracting tested hypotheses, which consist of related constructs, moderators or mediators, path coefficients, and p-values, from empirical studies using structural equation modeling (SEM). We combine state-of-the-art LLMs with a variety of post-processing measures to improve the relation extraction quality. An extensive evaluation yields recall scores of up to 79.2% in construct entity extraction, 58.4% in construct-mediator/moderator-construct extraction, and 39.3% in extracting the full tested hypotheses. We provide a manually annotated dataset of 72 SEM articles and 749 construct relations to facilitate future research. Our findings offer critical insights and suggest promising directions for advancing the field of automated construct relation extraction from scholarly documents.
https://aisel.aisnet.org/hicss-58/ks/ai_based_assistants/3