Videos have become ubiquitous on the Internet. And video analysis can provide lots of information for detecting and recognizing objects as well as help people understand human actions and interactions with the real world. However, facing data as huge as terabyte (TB) level, effective methods should be applied. Recurrent neural network (RNN) architecture has wildly been used on many sequential learning problems such as Language Model, Time-Series Analysis, etc. In this paper, we propose some variations of RNN such as stacked bidirectional Long Short Term Memory (LSTM) Gated Recurrent Unit (GRU) network with attention mechanism to categorize large-scale video data. We also explore different multimodal fusion methods. Our model combines both visual and audio information on both video and frame level and received great result. Ensemble methods are also applied. Because of its multimodal characteristics, we decide to call this method Deep Multimodal Learning (DML). Our DML-based model was trained on Google Cloud and our own server and was tested in a well-known video classification competition on Kaggle held by Google.
Multimodal is an emerging area which focuses on using various modalities to improve the performance of the model. More specifically, it tends to solve these five problems: Representation, Translation, Alignment, Fusion and Co-learning. Representation means learning how to represent and summarize multimodal data that exploits the complementarity and redundancy of multiple modalities. Translation studies how to translate data from one modality to another. Alignment tries to identify the direct relations between elements from over two modalities. Fusion is a challenge joining information from two or more modalities to perform a prediction. Co-learning transfers knowledge between modalities, their representation, and their predictive models @cite_8 . What we concern in our work is Fusion, cause the whole task is about prediction with two modalities: visual and audio. It includes two types of methods: Joint and Coordinated. We mainly cares about Joint method, which consists of three types of methods, including neural networks, graphical models and sequential. We care about the first one because our model adopts end-to-end neural networks.
@cite_8
2953039737
This paper proposes a novel multimodal fusion approach, aiming to produce best possible decisions by integrating information coming from multiple media. While most of the past multimodal approaches either work by projecting the features of different modalities into the same space, or by coordinating the representations of each modality through the use of constraints, our approach borrows from both visions. More specifically, assuming each modality can be processed by a separated deep convolutional network, allowing to take decisions independently from each modality, we introduce a central network linking the modality specific networks. This central network not only provides a common feature embedding but also regularizes the modality specific networks through the use of multi-task learning. The proposed approach is validated on 4 different computer vision tasks on which it consistently improves the accuracy of existing multimodal fusion approaches.