Multimodal Frame-Scoring Transformer for Video Summarization
As the number of video content has mushroomed in recent years, automatic
video summarization has come useful when we want to just peek at the content of
the video. However, there are two underlying limitations in generic video
summarization task. First, most previous approaches read in just visual
features as input, leaving other modality features behind. Second, existing
datasets for generic video summarization are relatively insufficient to train a
caption generator and multimodal feature extractors. To address these two
problems, this paper proposes the Multimodal Frame-Scoring Transformer (MFST)
framework exploiting visual, text and audio features and scoring a video with
respect to frames. Our MFST framework first extracts each modality features
(visual-text-audio) using pretrained encoders. Then, MFST trains the multimodal
frame-scoring transformer that uses video-text-audio representations as inputs
and predicts frame-level scores. Our extensive experiments with previous models
and ablation studies on TVSum and SumMe datasets demonstrate the effectiveness
and superiority of our proposed method.