Abstract

The current trends in automatic emotion recognition encompass the application of deep learning techniques, as, if applied to a multimodal approach, give the most promising results. The study presented in the paper follows this trend - the objective of the research is to propose a deep learning-based solution allowing to recognize emotions in circumplex model with performance metrics on a par with the ones achieved by competitive solutions. The observation channels used are physiological signals i.e. electrocardiography, electroencephalography and electroder- mal activity, while the applied technique is late fusion with Graph and Convolutional Neural Networks. The solution is validated for the AMIGOS dataset and the achieved results are com- parable to the baseline methods. While already satisfactory, the results still leave a place for further investigations.

Recommended Citation

Wiercinski, T. & Zawadzka, T. (2023). Late Fusion Approach for Multimodal Emotion Recognition Based on Convolutional and Graph Neural Networks. In A. R. da Silva, M. M. da Silva, J. Estima, C. Barry, M. Lang, H. Linger, & C. Schneider (Eds.), Information Systems Development, Organizational Aspects and Societal Trends (ISD2023 Proceedings). Lisbon, Portugal: Instituto Superior Técnico. ISBN: 978-989-33-5509-1. https://doi.org/10.62036/ISD.2023.41

Paper Type

Short Paper

DOI

10.62036/ISD.2023.41

Additional Files
References_DOI_ISD.2023.41.pdf (35 kB)
Download

Share

COinS
 

Late Fusion Approach for Multimodal Emotion Recognition Based on Convolutional and Graph Neural Networks

The current trends in automatic emotion recognition encompass the application of deep learning techniques, as, if applied to a multimodal approach, give the most promising results. The study presented in the paper follows this trend - the objective of the research is to propose a deep learning-based solution allowing to recognize emotions in circumplex model with performance metrics on a par with the ones achieved by competitive solutions. The observation channels used are physiological signals i.e. electrocardiography, electroencephalography and electroder- mal activity, while the applied technique is late fusion with Graph and Convolutional Neural Networks. The solution is validated for the AMIGOS dataset and the achieved results are com- parable to the baseline methods. While already satisfactory, the results still leave a place for further investigations.