Self Supervised Learning. Many recent works have used vision Transformers for self-supervised learning [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. Here we show a proof-of-concept that introducing a temporal latent bottleneck in Vision Transformers results in better self-supervised representations. We consider the SiT model from [2]} for this experiment. They use 3 objectives to pretrain their model - (1) The Reconstruction Objective - Reconstructs the input image, (2) The Rotation Prediction Objective - Predicts the rotation angle from [\(0^{\circ }\) , \(90^{\circ }\) , \(180^{\circ }\) , \(270^{\circ }\) ], and (3) The Constrastive Objective (similar to SimCLR [8]}). For the proposed approach, we introduce a temporal latent bottleneck into SiT resulting in the SiT + TLB model. SiT also uses additional trainable contrastive and rotation tokens as input for calculating the contrastive and rotation objectives respectively. For SiT + TLB, we take the mean across the temporal latent bottleneck state vectors and use the resulting representation for computing the rotation and contrastive objectives. We use a chunk length of 20 for the SiT + TLB model. We use the CIFAR10 dataset [9]} for our experiments. We pretrain the model for 400 epochs and evaluate the pretrained model at different epochs using linear probing. We use the CIFAR10 dataset for linear probing as well. We present the results for this experiment in Figure REF . We can see that the proposed approach outperforms SiT in all cases thus showing the effectiveness of the proposed architecture for self-supervised learning. For additional details regarding the setup, we refer the reader to Appendix section REF .
