In this paper, to take advantage of the desired properties of self-supervised learning, we propose a new video-based FER solution using contrastive learning. Our proposed method, Spatiotemporal Contrastive Learning of Representations (ST-CLR), first uses a novel temporal sampling scheme (temporal augmentation) to pick individual frames from the unlabeled input video to construct a sub-video. This will allow for various sub-videos to be constructed from any given input, resulting in better generalization. It then performs standard spatial augmentations, namely random cropping, color distortion, and random flipping to learn by maximizing the agreement between positive examples (augmentations on the same video) and minimizing the agreement for the negative pairs. A visual representation of our approach is shown in Figure REF . The figure demonstrates how the sub-video samples are projected onto a normalized embedding space, followed by minimizing of the distance between the positive pairs while maximizing the distance between the negative samples vs. the positives. We integrate this technique into a deep neural network that uses a R(2+1)dimensional framework instead of 3D convolutions to learn spatiotemporal representations. We perform extensive experiments and show the strong performance of our method on the Oulu-CASIA dataset [1]}, outperforming all the previous methods.
