Previous work has used patch-level contrastive training but limited its application to pre-training of a supervised model that was then fine tuned with labeled data [1]}. In that work, they combined a global contrastive loss and a local contrastive loss to learn both image representations and patch representations. On the other hand, other work has used a similar U-net structure but combined image-level and pixel-level representations [2]}. In both cases, notably, this was a pre-training step prior to supervised training rather than a standalone representation-learning method.
