The origin of contrastive learning can be traced back to the work by [1]} and [2]}.
This method has achieved outstanding success in self-supervised representation learning for images [3]}, [4]}, [5]}, [6]}, [7]}. The contrastive learning framework learns representations from unlabeled data by pulling positive pairs together and pushing negative pairs apart.
For training, the positive pairs are often obtained by taking two randomly augmented versions of the same sample and treating the other augmented examples within a mini-batch as negative examples [8]}, [6]}. The most widely used loss is the infoNCE loss (or contrastive loss) [4]}, [11]}, [6]}, [5]}.
Although many approaches adopt contrastive learning to represent sentences [14]}, [15]}, [16]}, it has so far not been applied
to word representations.
<TABLE>