Low-rank Multimodal Fusion Algorithm Based on Context Modeling

Abstract

As an important part of human daily life, video contains rich emotion information. Therefore, it is a current research trend to find efficient approaches to conducting emotional analysis on videos. Based on tensor fusion, we propose a low-rank multimodal fusion context modeling. At the beginning, modality information is preprocessed by GRU (Gate Recurrent Unit) in Recurrent Neural Network. We construct semantic dependencies to convey contextual information in the context of the video. The proposed model improves performance of applied emotion classification. Additionally, LMF (Low-rank Tensor Multimodal Fusion) with the advantage of end-to-end learning is implemented as a fusion mechanism to improve classification efficiency. We implemented the experiments on CMU-MOSI, POM, and IEMOCAP of multi-modal sentiment analysis, speaker traits and emotion recognition. And results show that our method improved the performance by a margin of 2.9%, 1.3%, and 12.2% respectively contrast with TFN (Tensor Fusion Network).