The reason why contrastive representation learning is biased towards the specific downstream tasks [1]}, [2]} is because the standard evaluation of SSL is based on the downstream tasks. This standard evaluation not only induces biases, but also is highly abstract since it is only appropriate when the features are linearly separable. Thus, an intuitive solution is to evaluate the learned representation differently.
<FIGURE>