We present a deep generative model that learns disentangled static and dynamic representations of data from unordered input. Our approach exploits regularities in sequential data that exist regardless of the order in which the data is viewed. The result of our factorized graphical model is a well-organized and coherent latent space for data dynamics. We demonstrate our method on several synthetic dynamic datasets and real video data featuring various facial expressions and head poses.
Unsupervised learning of disentangled representations can be related to modeling context or hierarchical structure in datasets. In particular, our approach invites comparison to the neural statistician'' of @cite_0 , whose context variable closely corresponds to our static encoding, although our model has a different dependence structure. On sequential data, @cite_5 propose a factorized hierarchical variational auto-encoder using a lookup table for different means, while @cite_8 condition a component of the factorized prior on the full ordered sequence. @cite_3 use an adversarial loss to factor the latent representation of a video frame in a stationary and temporally varying component. @cite_7 introduce a GAN that produces video clips by sequentially decoding a sample vector that consists of two parts: a sample from the motion subspace and a sample from a content subspace. In video generation, other directions can also be explored by decomposing the learned representation into deterministic and stochastic ( @cite_1 ).
@cite_7 @cite_8 @cite_1 @cite_3 @cite_0 @cite_5
2963045453 2619034550 2950662112 2795593503
We present a new model DRNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluating our approach on a range of synthetic and real videos. For the latter, we demonstrate the ability to coherently generate up to several hundred steps into the future. We present a new model DrNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future. The key idea behind the unsupervised learning of disentangled representations is that real-world data is generated by a few explanatory factors of variation which can be recovered by unsupervised learning algorithms. In this paper, we provide a sober look at recent progress in the field and challenge some common assumptions. We first theoretically show that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data. Then, we train more than 12000 models covering most prominent methods and evaluation metrics in a reproducible large-scale experimental study on seven different data sets. We observe that while the different methods successfully enforce properties encouraged'' by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Furthermore, increased disentanglement does not seem to lead to a decreased sample complexity of learning for downstream tasks. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision, investigate concrete benefits of enforcing disentanglement of the learned representations, and consider a reproducible experimental setup covering several data sets. We present a framework for learning disentangled and interpretable jointly continuous and discrete representations in an unsupervised manner. By augmenting the continuous latent distribution of variational autoencoders with a relaxed discrete distribution and controlling the amount of information encoded in each latent unit, we show how continuous and categorical factors of variation can be discovered automatically from data. The learned model also contains an inference network which can infer quantities such as angle and width of objects from image data in a completely unsupervised manner. Our experiments show that the framework disentangles continuous and discrete generative factors on various datasets, including disentangling digit type from stroke thickness, angle and width on MNIST, chair type from azimuth and width on the Chairs dataset and age from azimuth on CelebA.