[1]} is designed explicitly for vision transformer based on V1 and V2 [2]}, [3]}. Each image has two different versions generated by data augmentation and the images of different versions are fed into two different encoders. The model then learns to reduce the difference between the two identical images and increases the distance of those negative samples, where the difference is measured by InfoNCE [4]}.
Other modifications are 1) removing the memory queue, which is able to reduce the requirement of a substantial batch size, and 2) adopting the symmetrized loss. Suppose the first encoder outputs \(q_1, q_2\)  and the second encoder outputs \(k_1, k_2\) , the symmetrized loss \(\mathcal {L}\)  is computed by:
\(\mathcal {L} = \textit {infoNCE}(q_1, k_2) + \textit {infoNCE}(q_2, k_1).\) 
