In the self-supervised setting, the contrastive loss is applied to the outputs of a shallow MLP (the `projection network') that takes the encoder output as input [1]}. Our initial hypothesis is that it may be feasible to disentangle
the encoder outputs while training the projection network. We explore three methods for disentanglement
with an unsupervised contrastive loss: minimizing MI, orthogonalizing to encourage independence, and reducing dependencies via Hessian regularization on the contrastive loss. While each regularizer has supporting intuition, in practice we find that issues arise due to disentanglement on undesired axes, initialization sensitivity, or optimization. This prevents the regularizers from reliably scaling to real-world datasets.
We hope our results can be used to guide the community at large and spur new research directions.
