Much recent research has been devoted to video prediction and generation,  but mostly for short-scale time horizons. The hierarchical video prediction method by Villegas et al. (2017) is an example of a state of the art method for long term video prediction.   However, their method has limited applicability in practical settings as it requires a ground truth pose (e.g.,  poses of joints of a human) at training time.    This paper presents a long term hierarchical video prediction model that does not have such a restriction. We show that the network learns its own higher-level structure (e.g., pose equivalent hidden variables) that works better in cases where the ground truth pose does not fully capture all of the information needed to  predict  the  next  frame.    This  method  gives  sharper  results  than  other  video prediction methods which do not require a ground truth pose, and its efficiency is shown on the Humans 3.6M and Robot Pushing datasets. It is hypothesized that learning to predict the future and the effect of their actions is an important quality for intelligent agents that interact with their environment. This is a complicated task, as typical use cases require predicting the outcome of interactions between the agent and objects over multiple timesteps.In this work we are looking at the task of predicting the pixels of future video frames given the first few observed frames. We also consider the action conditional setting, in which we are given the action that the agent is taking and are tasked to predict the pixel level outcome of that action in the future.The method of BID20 is a novel way to generate long term video predictions, but requires ground truth human pose annotations. In this work we explore ways to generate videos using a hierarchical model without requiring a ground truth pose or other high level structure annotations for each frame. The method is hierarchical in the sense that it learns to generate a high level structure, then makes next frame predictions based on that structure. On datasets where the pose does not capture all of the information needed to predict future frames, letting the network define its own high level structure in addition to the pose is an improvement upon a BID20 . The EPEV method generates sharper images than BID3 on non deterministic datasets, and can generate further into the future on a toy dataset that we introduced. We posit an adversarial loss between the predictor and encoder would likely help with potentially uncertain scenarios and would fix the problem of the EPEV method sometimes generating blurry images,Under review as a conference paper at ICLR 2018 To see what the encoder has learned in the EPEV method, we can obtain results from the visual analogy network given the input from the encoder. The encoder is given the ground truth image.The results are shown in figure 9 . The results show that the encoder encodes where the person is Figure 9 : Results from the EPEV approach when the VAN is given the output of the encoder on the ground truth frame.in the image, as well as the orientation of the arms, legs and head to some extent. The results are not as good as one would expect from an autoencoder, since the encoder has the constraint that the encoding also has to be easy to predict.
We show ways to train a hierarchical video prediction model without needing pose labels.
{"src": "scitldr-AIC_train", "paper_id": "rkmtTJZCb"}