This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more. Recently there has been a lot of progress in self-supervised representation learning for textual sequences, followed by supervised fine-tuning (using small labeled datasets) of shallow (often linear) decoders on various downstream NLP tasks, such as sentiment classification. In this paper, we build on this work and propose a new method for self-supervised representation learning for videos, optionally accompanied by speech transcripts generated by automatic speech recognition (ASR). We show that fine-tuning linear decoders together with our self-supervised video representations, can achieve state of the art results on various supervised tasks, including video classification, segmentation and captioning. Our approach builds on the popular BERT (Bidirectional Encoder Representations from Transformers) model (Devlin et al., 2018) for text. This uses the Transformer architecture (Vaswani et al., 2017) to encode long sentences, and trains the model using the "masked language modeling" (MLM) training objective, in which the model must predict the missing words given their bidirectional context. The MLM loss requires that each token in the sequence be discrete. The VideoBERT model of (Sun et al., 2019a) therefore applied vector quantization (VQ) to video frames before passing them (along with optional ASR tokens) to the BERT model. Unfortunately, VQ loses fine-grained information that is often critical for downstream tasks. More recently, several papers (e.g., VilBERT and LXMERT (Tan & Bansal, 2019) ) proposed to address this limitation by directly measuring the visual similarity between frames using pre-trained visual encoders. In this paper, we propose a way to train bidirectional transformer models on sequences of realvalued vectors (e.g., video frames), x 1:T , using noise contrastive estimation (NCE), without needing pre-trained visual encoders. We call our method "Contastive Bidirectional Transformer" or CBT. We also develop a method that combines x 1:T with an optional sequence of discrete tokens, y 1:T (e.g., derived from ASR). In contrast to the VideoBERT paper (Sun et al., 2019a) , we provide a "lightweight" way of combining these signals after training each modality separately. In particular, we propose a cross-modal transformer to maximize the mutual information between x 1:T and y 1:T at the sequence level (rather than at the frame level). This method is robust to small misalignments between the sequences (e.g., if the words at time t do not exactly correspond to what is visually present in frame t). We demonstrate the effectiveness of the proposed approach for learning short-term visual representations, as well as longer term temporal representations. For visual representations, we encode each window of K frames using a 3D convolutional neural network S3D (Xie et al., 2018) , and then pass this sequence of features to the CBT model for self-supervised pretraining with the NCE loss on the We have shown how to extend the BERT model to learn representations from video in a self-supervised way, without needing vector quantization or pre-trained visual features. We have also shown how to extend this to the cross-modal setting, when ASR is available. Finally, we demonstrated that our method learns features that are far more useful than existing self-supervised methods for a variety of downstream video tasks, such as classification, captioning and segmentation. We believe that the simplicity and modularity of our method will let us scale to much larger unlabeled video datasets, which we hope will let us finally surpass supervised video pretraining (e.g., on Kinetics), just as other methods (e.g., CPC++ (Hénaff et al., 2019) ) have recently surpassed supervised image pretraining (on ImageNet).
Generalized BERT for continuous and cross-modal inputs; state-of-the-art self-supervised video representations.
{"src": "scitldr-AIC_train", "paper_id": "rJgRMkrtDr"}