As a variant of transfer learning, self-supervised learning of speech or language representations has been proposed these last few years, for instance with the BERT system [1]}, used for textual representation.
Such representations, computed by neural models trained on huge amounts of unlabeled data, have shown their effectiveness on some tasks under certain conditions, for instance for computer vision [2]} and Natural Language Processing (NLP) tasks such as Single-Sentence Classification, Text Similarity, Relevance Ranking [3]}, [4]}, [5]}, ASR [6]}, [7]}, or speech translation
 [8]}.
