Task 1: Masked Embedding Prediction (MEP). BERT [1]} is trained using the masked language model (MLM) task, which randomly selects input tokens and replaces them with a mask token. The model is then trained to predict the original (unmasked) tokens by solving a classification task with a cross-entropy loss. However, inputs to our model are real-valued audio-visual signals (rather than discrete tokens),In the form of RGB images and log-mel-scaled spectrograms. so applying the MLM task requires input discretization, which causes information loss [2]}, [3]}. We instead train our model to identify the correct visual clip or audio stream compared to a set of negative samples in a contrastive manner, which does not require input discretization.
