Contrastive learning and SSL are forms of unsupervised representation learning. In SSL, a supervisory signal is produced synthetically from the unlabeled data itself. These labels are used to formulate a pre-text task, and the network is trained on this task to learn efficient representations. In contrastive learning, a network is trained to sort representations in the latent space based on their similarity to one another. There have been many successful recent works on contrastive learning [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}. Two competing standout methods, which many works have built upon, are SimCLR[9]} and MoCo [4]}. SimCLR [9]} builds upon traditional contrastive learning, where positive and negative pairs are sorted, by simplifying the process, requiring no architectural or loss adjustments. MoCo, or momentum contrastive learning, builds dynamic dictionaries to perform dictionary look-up, where an encoded query should be similar to its matching key and dissimilar to others.
