To flexibly and efficiently reason about temporal sequences, abstract representations that compactly represent the important information in the sequence are needed. One way of constructing such representations is by focusing on the important events in a sequence. In this paper, we propose a model that learns both to discover such key events (or keyframes) as well as to represent the sequence in terms of them.   We do so using a hierarchical Keyframe-Inpainter (KeyIn) model that first generates keyframes and their temporal placement and then inpaints the sequences between keyframes. We propose a fully differentiable formulation for efficiently learning the keyframe placement. We show that KeyIn finds informative keyframes in several datasets with diverse dynamics. When evaluated on a planning task, KeyIn outperforms other recent proposals for learning hierarchical representations. When thinking about the future, humans focus their thoughts on the important things that may happen (When will the plane depart?) without fretting about the minor details that fill each intervening moment (What is the last word I will say to the taxi driver?). Because the vast majority of elements in a temporal sequence contains redundant information, a temporal abstraction can make reasoning and planning both easier and more efficient. How can we build such an abstraction? Consider the example of a lead animator who wants to show what happens in the next scene of a cartoon. Before worrying about every low-level detail, the animator first sketches out the story by keyframing, drawing the moments in time when the important events occur. The scene can then be easily finished by other animators who fill in the rest of the sequence from the story laid out by the keyframes. In this paper, we argue that learning to discover such informative keyframes from raw sequences is an efficient and powerful way to learn to reason about the future. Our goal is to learn such an abstraction for future image prediction. In contrast, much of the work on future image prediction has focused on frame-by-frame synthesis (Oh et al. (2015) ; Finn et al. (2016) ). This strategy puts an equal emphasis on each frame, irrespective of the redundant content it may contain or its usefulness for reasoning relative to the other predicted frames. Other recent work has considered predictions that "jump" more than one step into the future, but these approaches either used fixed-offset jumps (Buesing et al., 2018) or used heuristics to select the predicted frames (Neitz et al., 2018; Jayaraman et al., 2019; Gregor et al., 2019) . In this work, we propose a method that selects the keyframes that are most informative about the full sequence, so as to allow us to reason about the sequence holistically while only using a small subset of the frames. We do so by ensuring that the full sequence can be recovered from the keyframes with an inpainting strategy, similar to how a supporting animator finishes the story keyframed by the lead. One possible application for a model that discovers informative keyframes is in long-horizon planning. Recently, predictive models have been employed for model-based planning and control ). However, they reason about every single future time step, limiting their applicability to short horizon tasks. In contrast, we show that a model that reasons about the future using a small set of informative keyframes enables visual predictive planning for horizons much greater than previously possible by using keyframes as subgoals in a hierarchical planning framework. Figure 1: Keyframing the future. Instead of predicting one frame after the other, we propose to represent the sequence with the keyframes that depict the interesting moments of the sequence. The remaining frames can be inpainted given the keyframes. To discover informative frames in raw sequence data, we formulate a hierarchical probabilistic model in which a sequence is represented by a subset of its frames (see Fig. 1 ). In this two-stage model, a keyframing module represents the keyframes as well as their temporal placement with stochastic latent variables. The images that occur at the timepoints between keyframes are then inferred by an inpainting module. We parametrize this model with a neural network and formulate a variational lower bound on the sequence log-likelihood. Optimizing the resulting objective leads to a model that discovers informative future keyframes that can be easily inpainted to predict the full future sequence. Our contributions are as follows. We formulate a hierarchical approach for the discovery of informative keyframes using joint keyframing and inpainting (KEYIN). We propose a soft objective that allows us to train the model in a fully differentiable way. We first analyze our model on a simple dataset with stochastic dynamics in a controlled setting and show that it can reliably recover the underlying keyframe structure on visual data. We then show that our model discovers hierarchical temporal structure on more complex datasets of demonstrations: an egocentric gridworld environment and a simulated robotic pushing dataset, which is challenging for current approaches to visual planning. We demonstrate that the hierarchy discovered by KEYIN is useful for planning, and that the resulting approach outperforms other proposed hierarchical and non-hierarchical planning schemes on the pushing task. Specifically, we show that keyframes predicted by KEYIN can serve as useful subgoals that can be reached by a low-level planner, enabling long-horizon, hierarchical control. We presented KEYIN, a method for representing a sequence by its informative keyframes by jointly keyframing and inpainting. KEYIN first generates the keyframes of a sequence and their temporal placement and then produces the full sequence by inpainting between keyframes. We showed that KEYIN discovers informative keyframes on several datasets with stochastic dynamics. Furthermore, by using the keyframes for planning, we showed our method outperforms several other hierarchical planning schemes. Our method opens several avenues for future work. First, an improved training procedure that allows end-to-end training is desirable. Second, more powerful hierarchical planning approaches can be designed using the keyframe representation to scale to long-term real-world tasks. Finally, the proposed keyframing method can be applied to a variety of applications, including video summarization, video understanding, and multi-stage hierarchical video prediction.
We propose a model that learns to discover informative frames in a future video sequence and represent the video via its keyframes.
{"src": "scitldr-AIC_train", "paper_id": "BklfR3EYDH"}