Contrastive learning is a special pretext task to learn view-invariant representations through arbitrary data augmentations. It encourages different views of an image instance to have similar representations relative to negative samples [1]}, [2]}, [3]} or negative clusters [4]}, or even without using negatives [5]} at all.
Views of an image are commonly processed by a Siamese network architecture [6]} with a momentum encoder [2]} on one of its branches. To boost performance, the community has crafted various data augmentations including color jittering [1]}, Gaussian blurring [6]}, solarization [5]}, copy-and-paste [11]}, as well as determined their optimal hyper-parameters.
Contrastive models trained at scale are shown to perform on par with supervised learning [12]}, [13]}.
The contrastive learning framework has the flexibility to handle various data augmentations, while traditional pretext tasks need non-trivial engineering to be trained in a multi-task manner [14]}. Recently, there have been efforts [15]}, [16]} to combine contrastive learning with masked modeling objectives in a multi-task manner. The two tasks complement each other, but the intrinsic connection remains unclear.
