We build our model on top of SimCLRv2 [1]} for its simplicity and efficiency. It is composed of an encoder backbone network \(f_\theta \)  as well as a 3-layer contrastive head \(h_{\theta ^{\prime }}\) .
Given an in-distribution sample \(\mathcal {X}\) , a similarity function sim, and a distribution of training transformations \(\mathcal {T}_{train}\) , the goal is to simultaneously maximize \(\mathbb {E}_{x\sim \mathcal {X};t_0,t_1\sim \mathcal {T}_{train}}\left[sim(h_{\theta ^{\prime }}\circ f_\theta (t_0(x)),h_{\theta ^{\prime }}\circ f_\theta (t_1(x)))\right]\)  and minimize \(\mathbb {E}_{x,y\sim \mathcal {X};t_0,t_1\sim \mathcal {T}_{train}}\left[sim(h_{\theta ^{\prime }}\circ f_\theta (t_0(x)),h_{\theta ^{\prime }}\circ f_\theta (t_1(y)))\right]\) , i.e., we want to learn representations in which random transformations of a same example are close while random transformations of different examples are distant.
