Self-supervised learning has been proposed to reduce the reliance of learning techniques on output labels such as human annotations, by applying specific augmentations [1]} or transformations [2]}, [3]} to input data to generate pseudo-labels with which the model is trained.
Training of the model using the generated pseudo-labels can be seen as an auxiliary pre-training task, also sometimes referred to as a 'pretext' task. Successive to the pre-training step by means of the pretext tasks, main components of the model, for instance the convolutional blocks in a CNN, are generally frozen and transfer learning is used to develop another model for 'down-stream' tasks, i.e. the main aim classification problem [3]}. In this phase, the fully connected layers are often trained from scratch with the frozen convolutional blocks using the original labels in the dataset [1]}. In some cases, fine-tuning of the frozen components can also help achieve a better solution [6]}. This approach has been used on a variety of different problem domains including image analysis [6]}, [8]}, wearable-based activity recognition [9]}, [10]}, and affective computing with bio-signals [11]}, [12]}, [13]}.
