Paper Number
ICIS2025-2146
Paper Type
Short
Abstract
The quality of online health information can significantly influence public trust in healthcare decisions. Despite efforts like expert-driven reviews, large-scale, manual assessments have proven cost prohibitive and unsustainable. This study proposes an automated, scalable approach to evaluate health information quality using fine-tuned large language models (LLMs). Using an expert-annotated dataset sourced from HealthNewsReview.org, this study benchmarked several state-of-the-art LLMs, including open-weights models from Meta’s Llama and Alibaba’s Qwen series. Our best fine-tuned model demonstrated strong performance in accurately classifying information quality. By automating complex, criterion-based assessments that have traditionally been conducted by expert reviewers, this work provides a foundation for accessible and practical tools that help users evaluate health information, supporting informed public health decision-making.
Recommended Citation
Sang, Lan and Dobolyi, David, "Toward Trustworthy Health Information: Automated Quality Assessment of Health News Using Fine-Tuned LLMs" (2025). ICIS 2025 Proceedings. 14.
https://aisel.aisnet.org/icis2025/is_health/ishealthcare/14
Toward Trustworthy Health Information: Automated Quality Assessment of Health News Using Fine-Tuned LLMs
The quality of online health information can significantly influence public trust in healthcare decisions. Despite efforts like expert-driven reviews, large-scale, manual assessments have proven cost prohibitive and unsustainable. This study proposes an automated, scalable approach to evaluate health information quality using fine-tuned large language models (LLMs). Using an expert-annotated dataset sourced from HealthNewsReview.org, this study benchmarked several state-of-the-art LLMs, including open-weights models from Meta’s Llama and Alibaba’s Qwen series. Our best fine-tuned model demonstrated strong performance in accurately classifying information quality. By automating complex, criterion-based assessments that have traditionally been conducted by expert reviewers, this work provides a foundation for accessible and practical tools that help users evaluate health information, supporting informed public health decision-making.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
21-Healthcare