The objective function used in self-supervised learning techniques is the driving force behind the extraction of powerful speech representations.
In fact, the self-supervised objective has more impact on the learned representation than architectural differences between methods [1]}.
In Autoregressive Predictive Coding (APC) [2]}, [3]}, [4]}, [5]}, the objective is to predict a frame a few steps ahead, given the information up to that point. Another branch of research focuses on predicting the current frame given past and future context, by reconstructing several masked frames [6]}, [7]}, similar to the Masked Language Modeling (MLM) approach in Natural Language Processing [8]}. Finally, Contrastive Predictive Coding (CPC) [9]} is a popular technique in representation learning, where the objective is to predict the future in the latent space and a contrastive loss is applied to maximise mutual information. CPC has been successfully applied to speech recognition [10]}, [11]}, [12]}, [13]} and has shown to be able to learn robust and cross-lingual speech representations [14]}, [15]}, [16]}. We refer to the literature for other related work in self-supervised and unsupervised representation learning [17]}, [18]}, [19]}, [20]}, [21]}, [22]}, [23]}, [24]}.
