This review analyzed the key components in contrastive learning methods. First, we introduce that pretext invariant learning methods [1]} outperform pretext covariant learning methods because the former preserves the mutual visual semantics under image transformation. After that, based on pretext invariant learning, contrastive learning methods e.g. SimCLR [2]} used contrastive loss to achieve supervised-level representation learning performance, with a dependency on 1) large numbers of negative examples for contrasting; 2) a specific composition of data augmentation operations. In the following, we discuss works that tackle these two dependencies.
For negative examples, we introduced MoCHi [3]} as a possible method to understand and improve the selection of negative examples. For the specific composition of data augmentations, we discussed people's criticism [4]} about it and introduced two possible solutions: a general representation evaluation method [5]} and a covariant preserved pretext invariant learning method [6]}. With all these components being discussed, we hope this literature review could help people understand the contrastive learning and develop novel more methods to improve contrastive learning.
