Predicting the future in real-world settings, particularly from raw sensory observations such as images, is exceptionally challenging. Real-world events can be stochastic and unpredictable, and the high dimensionality and complexity of natural images requires the predictive model to build an intricate understanding of the natural world. Many existing methods tackle this problem by making simplifying assumptions about the environment. One common assumption is that the outcome is deterministic and there is only one plausible future. This can lead to low-quality predictions in real-world settings with stochastic dynamics. In this paper, we develop a stochastic variational video prediction (SV2P) method that predicts a different possible future for each sample of its latent variables. To the best of our knowledge, our model is the first to provide effective stochastic multi-frame prediction for real-world video. We demonstrate the capability of the proposed method in predicting detailed future frames of videos on multiple real-world datasets, both action-free and action-conditioned. We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods. Our SV2P implementation will be open sourced upon publication. Understanding the interaction dynamics of objects and predicting what happens next is one of the key capabilities of humans which we heavily rely on to make decisions in everyday life BID3 . A model that can accurately predict future observations of complex sensory modalities such as vision must internally represent the complex dynamics of real-world objects and people, and therefore is more likely to acquire a representation that can be used for a variety of visual perception tasks, such as object tracking and action recognition BID31 BID25 BID7 . Furthermore, such models can be inherently useful themselves, for example, to allow an autonomous agent or robot to decide how to interact with the world to bring about a desired outcome BID27 .However , modeling future distributions over images is a challenging task, given the high dimensionality of the data and the complex dynamics of the environment. Hence, it is common to make various simplifying assumptions. One particularly common assumption is that the environment is deterministic and that there is only one possible future BID5 BID31 BID1 BID25 . Models conditioned on the actions of an agent frequently make this assumption, since the world is more deterministic in these settings BID27 BID10 . However, most real-world prediction tasks, including the action-conditioned settings, are in fact not deterministic, and a deterministic model can lose many of the nuances that are present in real physical interactions. Given the stochastic nature of video prediction, any deterministic model is obliged to predict a statistic of all the possible outcomes. For example, deterministic models trained with a mean squared error loss function generate the expected value of all the possibilities for each pixel independently, which is inherently blurry BID26 . Figure 1: Importance of stochasticity in video prediction. In each video, a random shape follows a random direction (first row). Given only the first frame, the deterministic model from BID10 predicts the average of all the possibilities. The third row is the output of SV2P with latent sampled from approximated posterior which predicts the correct motion. Last two rows are stochastic outcomes using random latent values sampled from assumed prior. As observed, these outcomes are random but within the range of possible futures. Second sample of Figure 1c shows a case where the model predicts the average of more than one outcome.Our main contribution in this paper is a stochastic variational method for video prediction, named SV2P, that predicts a different plausible future for each sample of its latent random variables. We also provide a stable training procedure for training a neural network based implementation of this method. To the extent of our knowledge, SV2P is the first latent variable model to successfully predict multiple frames in real-world settings. Our model also supports action-conditioned predictions, while still being able to predict stochastic outcomes of ambiguous actions, as exemplified in our experiments. We evaluate SV2P on multiple real-world video datasets, as well as a carefully designed toy dataset that highlights the importance of stochasticity in video prediction (see Figure 1 ). In both our qualitative and quantitative comparisons, SV2P produces substantially improved video predictions when compared to the same model without stochasticity, with respect to standard metrics such as PSNR and SSIM. The stochastic nature of SV2P is most apparent when viewing the predicted videos. Therefore, we highly encourage the reader to check the project website https://goo.gl/iywUHc to view the actual videos of the experiments. The TensorFlow BID0 implementation of this project will be open sourced upon publication. We proposed stochastic variational video prediction (SV2P), an approach for multi-step video prediction based on variational inference. Our primary contributions include an effective stochastic prediction method with latent variables, a network architecture that succeeds on natural videos, and a training procedure that provides for stable optimization. The source code for our method will be released upon acceptance. We evaluated our proposed method on three real-world datasets in actionconditioned and action-free settings, as well as one toy dataset which has been carefully designed to highlight the importance of the stochasticity in video prediction. Both qualitative and quantitative results indicate higher quality predictions compared to other deterministic and stochastic baselines.SV2P can be expanded in numerous ways. First, the current inference network design is fully convolutional, which exposes multiple limitations, such as unmodeled spatial correlations between the latent variables. The model could be improved by incorporating the spatial correlation induced by the convolutions into the prior, using a learned structured prior in place of the standard spherical Gaussian. Time-variant posterior approximation to reflect the new information that is revealed as the video progresses, is another possible SV2P improvement. However, as discussed in Section 3, this requires incentivizing the inference network to incorporate the latent information at training time. This would allow time-variant latent distributions which is more aligned with generative neural models for time-series BID19 BID12 BID22 .Another exciting direction for future research would be to study how stochastic predictions can be used to act in the real world, producing model-based reinforcement learning methods that can execute risk-sensitive behaviors from raw image observations. Accounting for risk in this way could be especially important in safety-critical settings, such as robotics. In lack of actions and therefore high stochasticity, BID10 only blurs the robotic arm out while the proposed method predicts sharper frames on each sampling. SV2P also predicts the interaction dynamics between random movements of the arm and the objects. BID10 . This is mostly evident in zoomed in objects which have been pushed by the arm. Figure 10: Prediction results on the action-free Human3.6M dataset. SV2P predicts a different outcome on each sampling given the latent. In the left example, the model predicts walking as well as stopping which result in different outputs in predicted future frames. Similarly, the right example demonstrates various outcomes including spinning. BID29 with SV2P on the robotic pushing dataset. We use the same best PSNR out of 100 random samples for both methods. Besides stochastic movements of the pushed objects, another source of stochasticity is the starting lag in movements of the robotic arm. SV2P generates sharper images compared to BID10 (notice the pushed objects in zoomed images) with less noise compared to BID29 (look at the accumulated noise in later frames).A TRAINING DETAILS FIG2 contains details of the network architectures used as generative and inference models. In all of the experiments we used the same set of hyper-parameters which can be found in TAB1 . In the first step of training, we disable the inference network and instead sample latent values from N (0, I). In step 2, the latent values will be sampled from the approximated posterior q φ (z|x 0:T ) = N µ(x 0:T ), σ(x 0:T ) . Please note that the inference network approximates log(σ) instead of σ for numerical stability. To gradually switch from Step 2 of training procedure to Step 3, we increase β linearly from its starting value to its end value over the length of training.
Stochastic variational video prediction in real-world settings.
{"src": "scitldr-AIC_train", "paper_id": "rk49Mg-CW"}