Most of the earlier SSL methods are based on pretext-covariant learning [1]}, [2]}, [3]}, [4]}. Pretext-covariant learning methods aim to learn a representation that is covariant with certain properties of the pretext task's transformations (as mentioned in section ). For example, [4]} proposed a pretext task that predicts image rotation, where the images are augmented by rotating in certain degrees and the network is trained to distinguish the rotation angles. Thus, the network learns a representation that covaries with the rotation angle, and this is pretext covariant representation learning.
