Self-supervised learning with video data often uses the temporal information in the video sequence, for instance tracking the temporal order of the frames of a video [1]}, [2]}. The intuition here is that the movement of objects in frames can be tracked in a sequence of consecutive video frames, thus the representation of frames that are temporally close in the video must be similar. In [1]}, a self-supervised method was proposed that takes a set of consecutive frames of a video to use as the main input. This was followed by selecting two additional sets from the same video, where the one temporally closest to the main set being designated as the positive set, while the other being designated as the negative set. This approach learned the video representations by enforcing the embeddings obtained from of the two close sub-videos to be similar in the latent space, while forcing the embedding of the negative pair to be far apart in the latent space. In [2]}, the performance of [1]} was improved by picking the negative sub-video from a completely different video. In [6]}, a contrastive learning method was proposed for videos that, similar to [1]}, uses consecutive frames to form the positive and negative pairs.
