Paper Number
ICIS2025-1758
Paper Type
Complete
Abstract
Scam-related spam remains a significant societal concern due to its widespread negative impact. DeepSeek, a recently introduced open-source large language model, demonstrated strong capabilities in reasoning and problem-solving. However, its performance and susceptibility to misclassification in spam detection tasks have not been thoroughly evaluated. This study investigates DeepSeek’s applicability by comparing it with traditional AI and LLM models on SMS and email spam detection. Results show that DeepSeek, particularly DeepSeek-R1, outperforms baseline models with high accuracy (0.989 for SMS and 0.991 for email) and strong MCC Scores (0.946 for SMS and 0.970 for email). A linguistic analysis of scam texts reveals that DeepSeek effectively identifies assertive and commissive speech acts, enhancing spam detection. Nonetheless, it struggles with expressive acts and low-directive content, indicating areas for improvement. Despite these challenges, the findings offer valuable insights for refining detection systems and highlight DeepSeek’s potential in addressing evolving threats in spam detection.
Recommended Citation
Cai, Wei; Tiong, Goh; and Hong, Yvonne, "Seeking Deep Insights: Misclassification and Performance in DeepSeek’s Spam Detection" (2025). ICIS 2025 Proceedings. 6.
https://aisel.aisnet.org/icis2025/da_bus/da_bus/6
Seeking Deep Insights: Misclassification and Performance in DeepSeek’s Spam Detection
Scam-related spam remains a significant societal concern due to its widespread negative impact. DeepSeek, a recently introduced open-source large language model, demonstrated strong capabilities in reasoning and problem-solving. However, its performance and susceptibility to misclassification in spam detection tasks have not been thoroughly evaluated. This study investigates DeepSeek’s applicability by comparing it with traditional AI and LLM models on SMS and email spam detection. Results show that DeepSeek, particularly DeepSeek-R1, outperforms baseline models with high accuracy (0.989 for SMS and 0.991 for email) and strong MCC Scores (0.946 for SMS and 0.970 for email). A linguistic analysis of scam texts reveals that DeepSeek effectively identifies assertive and commissive speech acts, enhancing spam detection. Nonetheless, it struggles with expressive acts and low-directive content, indicating areas for improvement. Despite these challenges, the findings offer valuable insights for refining detection systems and highlight DeepSeek’s potential in addressing evolving threats in spam detection.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
07-DataAnalytics