Paper Number
ICIS2025-2168
Paper Type
Complete
Abstract
Disasters and urgent situations develop quickly, necessitating rapid response. Social media becomes a vital tool for coordination, but the overwhelming volume of posts impedes effective categorization. Large language models (LLMs) demonstrate strong capability to enhance classification of user-generated text for disaster management, yet their computational cost when introducing new knowledge remains a constraint. The dynamic nature of crisis information demands models that adapt to new incidents without costly retraining. To address this, we adopt the IS computational design science paradigm and propose DMEMIT, a distilled knowledge editing framework for LLMs targeting disaster-related tweet classification. Evaluation on three datasets (HumAID, CrisisNLP, CrisisLex) shows DMEMIT surpasses benchmarks, attaining up to 8% higher classification accuracy and an editing success score of 0.42 vs. 0.39 (MEMIT) and 0.34 (MEND). Explainability analysis yields 80% accuracy and 0.73 IOU, validating interpretability and predictions. Our research advances public IS by enabling AI-driven crisis infrastructures with efficient knowledge adaptation, addressing urgent societal needs.
Recommended Citation
Mukta, Md. Saddam Hossain and Islam, Najmul, "Distillation-Based Knowledge Editing of LLMs for Social Media Text Classification in Disaster Monitoring: A Design Science Approach" (2025). ICIS 2025 Proceedings. 11.
https://aisel.aisnet.org/icis2025/public_is/public_is/11
Distillation-Based Knowledge Editing of LLMs for Social Media Text Classification in Disaster Monitoring: A Design Science Approach
Disasters and urgent situations develop quickly, necessitating rapid response. Social media becomes a vital tool for coordination, but the overwhelming volume of posts impedes effective categorization. Large language models (LLMs) demonstrate strong capability to enhance classification of user-generated text for disaster management, yet their computational cost when introducing new knowledge remains a constraint. The dynamic nature of crisis information demands models that adapt to new incidents without costly retraining. To address this, we adopt the IS computational design science paradigm and propose DMEMIT, a distilled knowledge editing framework for LLMs targeting disaster-related tweet classification. Evaluation on three datasets (HumAID, CrisisNLP, CrisisLex) shows DMEMIT surpasses benchmarks, attaining up to 8% higher classification accuracy and an editing success score of 0.42 vs. 0.39 (MEMIT) and 0.34 (MEND). Explainability analysis yields 80% accuracy and 0.73 IOU, validating interpretability and predictions. Our research advances public IS by enabling AI-driven crisis infrastructures with efficient knowledge adaptation, addressing urgent societal needs.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
20-PublicIS