We propose a generative framework which takes on the video frame interpolation problem. Our framework, which we call Deep Locally Linear Embedding (DeepLLE), is powered by a deep convolutional neural network (CNN) while it can be used instantly like conventional models. DeepLLE fits an auto-encoding CNN to a set of several consecutive frames and embeds a linearity constraint on the latent codes so that new frames can be generated by interpolating new latent codes. Different from the current deep learning paradigm which requires training on large datasets, DeepLLE works in a plug-and-play and unsupervised manner, and is able to generate an arbitrary number of frames. Thorough experiments demonstrate that without bells and whistles, our method is highly competitive among current state-of-the-art models.
The technical inspiration for our work is Roweis and Saul @cite_4 . The method finds a linear relationship between neighboring samples in their original space and then finds another space, preferably having lower dimension, where this relationship is preserved. We, however, do not impose any linearity on images but only on their latent codes and this linear relationship is explicitly specified instead of learning from data as in @cite_4 . Another interpolation research close to ours is proposed by Bregler al which also relies on manifold learning @cite_22 . However, they estimate a non-linear manifold for the whole series of frames and the images are simply talking lips. Here, the method is designed to work with natural images containing different kinds of motion. In addition, our method does not estimate the manifold of the whole video but several consecutive frames only. In a remote study @cite_2 , an observation that latent variables can encode motions backs up our idea, but their latent codes are randomly sampled from a Gaussian distribution, and their work concentrates on predicting the trajectory of motions given a still image, and hence the generated images are nowhere near realistic.
@cite_4 @cite_22 @cite_2
2053186076 2952390294
Many areas of science depend on exploratory data analysis and visualization. The need to analyze large amounts of multivariate data raises the fundamental problem of dimensionality reduction: how to discover compact representations of high-dimensional data. Here, we introduce locally linear embedding (LLE), an unsupervised learning algorithm that computes low-dimensional, neighborhood-preserving embeddings of high-dimensional inputs. Unlike clustering methods for local dimensionality reduction, LLE maps its inputs into a single global coordinate system of lower dimensionality, and its optimizations do not involve local minima. By exploiting the local symmetries of linear reconstructions, LLE is able to learn the global structure of nonlinear manifolds, such as those generated by images of faces or documents of text. How do we judge similarity? Our mental representations of the world are formed by processing large numbers of sensory in In a given scene, humans can often easily predict a set of immediate future events that might happen. However, generalized pixel-level anticipation in computer vision systems is difficult because machine learning struggles with the ambiguity inherent in predicting the future. In this paper, we focus on predicting the dense trajectory of pixels in a scene, specifically what will move in the scene, where it will travel, and how it will deform over the course of one second. We propose a conditional variational autoencoder as a solution to this problem. In this framework, direct inference from the image shapes the distribution of possible trajectories, while latent variables encode any necessary information that is not available in the image. We show that our method is able to successfully predict events in a wide variety of scenes and can produce multiple different predictions when the future is ambiguous. Our algorithm is trained on thousands of diverse, realistic videos and requires absolutely no human labeling. In addition to non-semantic action prediction, we find that our method learns a representation that is applicable to semantic vision tasks.