Paper Type
Complete
Abstract
Study presents a comprehensive evaluation of a mental health language model, fine-tuned on a curated dataset of questions and conversational mental health data across text and preliminary audio modalities. The dataset comprises structured information from reputable mental health blogs, conversational data from therapist-patient interactions, and audio records. We leveraged the GPT-4 model as a reference against fine-tuned GPT-2, TinyLLaMA, DialoGPT and BART. Application of retrieval-augmented generation (RAG) pipeline and an audio-based emotion recognition module using HuBERT-Large-SUPERB-ER model to explore multimodal conditioning via inferred emotions (e.g. neutral, anger, happy, sad). Evaluation metrics included linguistic analysis (readability, sentiment, response diversity) and semantic similarity (ROUGE-L, BLEU, BERTScore), comparing the model’s responses with those from GPT-4. Our findings highlight the model’s capability to deliver accurate, empathetic, and user-centered mental health support, demonstrating its suitability for addressing a wide range of mental health inquiries. We plan to further improve the model's accuracy in this area.
Paper Number
1840
Recommended Citation
Vaishnav, Lakshika; Jonnala, Ram Gopal Reddy; and Goel, Sanjay, "EVALUATING FINE-TUNED LLMS FOR MENTAL HEALTH SUPPORT: TEXT AND AUDIO MODALITIES" (2026). AMCIS 2026 Proceedings. 9.
https://aisel.aisnet.org/amcis2026/ai_aiaa/ai_aiaa/9
EVALUATING FINE-TUNED LLMS FOR MENTAL HEALTH SUPPORT: TEXT AND AUDIO MODALITIES
Study presents a comprehensive evaluation of a mental health language model, fine-tuned on a curated dataset of questions and conversational mental health data across text and preliminary audio modalities. The dataset comprises structured information from reputable mental health blogs, conversational data from therapist-patient interactions, and audio records. We leveraged the GPT-4 model as a reference against fine-tuned GPT-2, TinyLLaMA, DialoGPT and BART. Application of retrieval-augmented generation (RAG) pipeline and an audio-based emotion recognition module using HuBERT-Large-SUPERB-ER model to explore multimodal conditioning via inferred emotions (e.g. neutral, anger, happy, sad). Evaluation metrics included linguistic analysis (readability, sentiment, response diversity) and semantic similarity (ROUGE-L, BLEU, BERTScore), comparing the model’s responses with those from GPT-4. Our findings highlight the model’s capability to deliver accurate, empathetic, and user-centered mental health support, demonstrating its suitability for addressing a wide range of mental health inquiries. We plan to further improve the model's accuracy in this area.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIG AIAA