Large scale video classification using both visual and audio features on YouTube 8 M dataset
Convolutional Neural Networks (CNNs) have achieved state of art in image classification and have been progressing rapidly in the field of video classification and audio understanding in recent years. Encouraged by these results, we propose jointly using video and audio features in order to provide a promising classifier for the Youtube 8M Kaggle challenge a video classification task for a dataset of 7 million YouTube videos belonging to 4716 classes[2] In this paper, we explore several models of different combination of video level visual and audio features. Our best model uses the mixture of experts (MoE) to receive three inputs. They are video level visual feature only, audio feature only and a concatenation of visual and audio feature. Then we apply a dense layer followed by a ReLu activation layer. The gating network we applied in MoE uses softmax function. We train our model with Tensorflow using 1 GPU on Google Cloud Platform. The average training time is 1 hour 15 minutes. As a result, we achieve Avg Hit@1 of 0.84, Avg PERR (average precision at equal recall rate) of 0.709, and mAP (mean average precision) of 0.415 compared to the best performing baseline proposed in [2] with Avg Hit@1 0.645, Avg PERR of 0.573, mAP of 0.266.