Temporal Alignment Loss. Taking inspiration from prior works on video representation learning [1]}, [2]}, [3]}, [4]}, [5]}, we employ the task of temporal alignment for learning task-specific representations. Given a pair of videos, the task of self-supervised alignment implicitly assumes that there exists true semantic correspondence between the two sequences, i.e., both videos share a common semantic space. These works have shown that optimizing for alignment leads to representations that could be used for tasks that require understanding task progression such as action-classification. This is because in order to solve for alignment, a learning model has to learn features that are (1) common across most videos and (2) exhibit temporal ordering. For a sufficiently large dataset with single task, the most common visual features would be distinct phases of a task that appear in all videos and if the task has small permutations, these distinct features would also exhibit temporal order. In such scenarios, the representations learned by optimizing for alignment are task-specific and invariant to changes in viewpoints, appearances and actor embodiments.
