We introduce a self-supervised method for learning visual correspondence from unlabeled video. The main idea is to use cycle-consistency in time as free supervisory signal for learning visual representations from scratch. At training time, our model learns a feature map representation to be useful for performing cycle-consistent tracking. At test time, we use the acquired representation to find nearest neighbors across space and time. We demonstrate the generalizability of the representation -- without finetuning -- across a range of visual correspondence tasks, including video object segmentation, keypoint tracking, and optical flow. Our approach outperforms previous self-supervised methods and performs competitively with strongly supervised methods.
Temporal structure serves as a useful signal for learning because the visual world is continuous and smoothly-varying. Spatio-temporal stability is thought to play a crucial role in the development of invariant representations in biological vision @cite_31 @cite_62 @cite_55 @cite_29 . For example, Wood @cite_67 showed that for newborn chicks raised in a visual world that was not temporally smooth, object recognition abilities were severely impaired. Computational approaches for unsupervised learning have sought to leverage this continuity, such as continuous transformation learning @cite_58 @cite_87 , slow" feature learning @cite_38 @cite_79 @cite_63 and information maximization between neighbouring patches in time @cite_35 . Our work can be seen as slow feature learning with fixation, learned end-to-end without supervision.
@cite_38 @cite_67 @cite_62 @cite_35 @cite_87 @cite_29 @cite_55 @cite_79 @cite_63 @cite_31 @cite_58
2401964427 2964037671 1650052828 2743563068
Abstract Understanding how the brain learns to recognize objects is one of the ultimate goals in the cognitive sciences. To date, however, we have not yet characterized the environmental factors that cause object recognition to emerge in the newborn brain. Here, I present the results of a high-throughput controlled-rearing experiment that examined whether the development of object recognition requires experience with temporally smooth visual objects. When newborn chicks ( Gallus gallus ) were raised with virtual objects that moved smoothly over time, the chicks developed accurate color recognition, shape recognition, and color-shape binding abilities. In contrast, when newborn chicks were raised with virtual objects that moved non-smoothly over time, the chicks’ object recognition abilities were severely impaired. These results provide evidence for a “smoothness constraint” on newborn object recognition. Experience with temporally smooth objects facilitates the development of object recognition. We present an unsupervised representation learning approach using videos without semantic labels. We leverage the temporal coherence as a supervisory signal by formulating representation learning as a sequence sorting task. We take temporally shuffled frames (i.e., in non-chronological order) as inputs and train a convolutional neural network to sort the shuffled sequences. Similar to comparison-based sorting algorithms, we propose to extract features from all frame pairs and aggregate them to predict the correct order. As sorting shuffled image sequence requires an understanding of the statistical temporal structure of images, training with such a proxy task allows us to learn rich and generalizable visual representation. We validate the effectiveness of the learned representation using our method as pre-training on high-level recognition problems. The experimental results show that our method compares favorably against state-of-the-art methods on action recognition, image classification, and object detection tasks. Representations that are invariant to translation, scale and other transformations, can considerably reduce the sample complexity of learning, allowing recognition of new object classes from very few examples|a hallmark of human recognition. Empirical estimates of one-dimensional projections of the distribution induced by a group of ane transformations are proven to represent a unique and invariant signature associated with an image. We show how projections yielding invariant signatures for future images can be learned automatically, and updated continuously, during unsupervised visual experience. A module performing ltering and pooling, like simple and complex cells as proposed by Hubel and Wiesel, can compute such estimates. Under this view, a pooling stage estimates a one-dimensional probability distribution. Invariance from observations through a restricted window is equivalent to a sparsity property w.r.t. to a transformation, which yields templates that are a) Gabor for optimal simultaneous invariance to translation and scale or b) very specic for complex, class-dependent transformations such as rotation in depth of faces. Hierarchical architectures consisting of this basic Hubel-Wiesel module inherit its properties of invariance, stability, and discriminability while capturing the compositional organization of the visual world in terms of wholes and parts, and are invariant to complex transformations that may only be locally ane. The theory applies to several existing deep learning convolutional architectures for image and speech recognition. It also suggests that the main computational goal of the ventral stream of visual cortex is to provide a hierarchical representation of new objects images which is invariant to transformations, stable, and discriminative for recognition|and that this representation may be continuously learned in an unsupervised way during development and natural visual experience. We present an unsupervised representation learning approach using videos without semantic labels. We leverage the temporal coherence as a supervisory signal by formulating representation learning as a sequence sorting task. We take temporally shuffled frames (i.e., in non-chronological order) as inputs and train a convolutional neural network to sort the shuffled sequences. Similar to comparison-based sorting algorithms, we propose to extract features from all frame pairs and aggregate them to predict the correct order. As sorting shuffled image sequence requires an understanding of the statistical temporal structure of images, training with such a proxy task allows us to learn rich and generalizable visual representation. We validate the effectiveness of the learned representation using our method as pre-training on high-level recognition problems. The experimental results show that our method compares favorably against state-of-the-art methods on action recognition, image classification and object detection tasks.