In SimCLR, the authors conducted a set of ablation experiments to examine the key components of the success of the learning, as we summarized below. 1) The specific composition of challenging data augmentation operations is crucial to learning good representations.
Otherwise, if an 'easier' set of data augmentation is used, the model could almost perfectly identify the positive pairs in the contrastive task while still learn a bad representation. 2) A larger batch size with more negative examples would significantly benefit the training. This observation is in line with other SSL papers [1]}, [2]}, [3]} using contrastive loss and the analysis of contrastive loss. 3) The hidden layer before the projection head is a better representation than the layer after, which means that \(\mathbf {z}\)  is a better representation than \(\mathbf {h}\) . It is conjectured that this phenomenon is due to the information loss induced by the contrastive loss, as in pretext covariant networks, the penultimate layer representation is always better than the final layer representations. The above three key components of the SimCLR together helped SimCLR and other SSL methods to achieve supervised learning level performances.
