In particular, recent works on self-supervised learning advance to contrast between images to perform representation learning by leveraging the combination of various transformations, called contrastive learning [1]}, [2]}, [3]}, [4]}, [5]}. For instance, SimCLR [1]} as a representative work first generates a pair of positive views by applying a sequence of transformations to an image twice,
then it learns image representation by attracting the positive pairs and pushing negative views from other images away.

SimSiam [3]} can be thought of as “SimCLR without negatives”, it introduces a predictor network in its forward process on one view and applies a stop-gradient operation in its backward process on the other view. Moreover, as it uses neither the negative sample pairs nor the momentum encoders (what other self-supervised methods, e.g. MoCO [8]} and BYOL [2]}, would need),
it has smaller model size during training (compared to MoCO and BYOL) as well as better support for the small training batch. SimSiam thus becomes suitable for the computational-cost-sensitive problems such as OCL, the main topic of this paper.
