Value functions are crucial for model-free Reinforcement Learning (RL) to obtain a policy implicitly or guide the policy updates. Value estimation heavily depends on the stochasticity of environmental dynamics and the quality of reward signals. In this paper, we propose a two-step understanding of value estimation from the perspective of future prediction, through decomposing the value function into a reward-independent future dynamics part and a policy-independent trajectory return part. We then derive a practical deep RL algorithm from the above decomposition, consisting of a convolutional trajectory representation model, a conditional variational dynamics model to predict the expected representation of future trajectory and a convex trajectory return model that maps a trajectory representation to its return. Our algorithm is evaluated in MuJoCo continuous control tasks and shows superior results under both common settings and delayed reward settings.
Thinking about the future has been considered as an integral component of human cognition @cite_8 @cite_29 . In neuroscience, one concept named the prospective brain @cite_28 indicates that a crucial function of the brain is to use stored information to predict possible future events. The idea of future prediction is also studied in model-based RL @cite_26 @cite_0 . Simulated Policy Learning @cite_6 is proposed to learn one-step predictive models from the real environment and then train a policy within the simulated environment. Multi-steps and long-term future are also modeled in @cite_21 @cite_24 with recurrent variational dynamics models, after which actions are chosen through online planning with Model-Predictive Control (MPC). Besides, another related work is @cite_12 , in which a supervised model is trained to predict the residuals of goal-related measurements at a set of temporal offsets in the future. With a manually designed goal vector, actions are chosen to maximize the predicted outcomes.
@cite_26 @cite_8 @cite_28 @cite_29 @cite_21 @cite_6 @cite_0 @cite_24 @cite_12
1540685400 2104128137 2095047274 2160623437 2950004691 2920362155 1980035368 2918394844 2553701371
Abstract Thinking about the future is an integral component of human cognition – one that has been claimed to distinguish us from other species. Building on the construct of episodic memory, we introduce the concept of ‘episodic future thinking': a projection of the self into the future to pre-experience an event. We argue that episodic future thinking has explanatory value when considering recent work in many areas of psychology: cognitive, social and personality, developmental, clinical and neuropsychology. Episodic future thinking can serve as a unifying concept, connecting aspects of diverse research findings and identifying key questions requiring further reflection and study. A rapidly growing number of recent studies show that imagining the future depends on much of the same neural machinery that is needed for remembering the past. These findings have led to the concept of the prospective brain; an idea that a crucial function of the brain is to use stored information to imagine, simulate and predict possible future events. We suggest that processes such as memory can be productively re-conceptualized in light of this idea. Episodic memory is widely conceived as a fundamentally constructive, rather than reproductive, process that is prone to various kinds of errors and illusions. With a view towards examining the functions served by a constructive episodic memory system, we consider recent neuropsychological and neuroimaging studies indicating that some types of memory distortions reflect the operation of adaptive processes. An important function of a constructive episodic memory is to allow individuals to simulate or imagine future episodes, happenings and scenarios. Since the future is not an exact repetition of the past, simulation of future episodes requires a system that can draw on the past in a manner that flexibly extracts and recombines elements of previous experiences. Consistent with this constructive episodic simulation hypothesis, we consider cognitive, neuropsychological and neuroimaging evidence showing that there is considerable overlap in the psychological and neural processes involved in remembering the past and imagining the future. Planning has been very successful for control tasks with known environment dynamics. To leverage planning in unknown environments, the agent needs to learn the dynamics from interactions with the world. However, learning dynamics models that are accurate enough for planning has been a long-standing challenge, especially in image-based domains. We propose the Deep Planning Network (PlaNet), a purely model-based agent that learns the environment dynamics from images and chooses actions through fast online planning in latent space. To achieve high performance, the dynamics model must accurately predict the rewards ahead for multiple time steps. We approach this using a latent dynamics model with both deterministic and stochastic transition components. Moreover, we propose a multi-step variational inference objective that we name latent overshooting. Using only pixel observations, our agent solves continuous control tasks with contact dynamics, partial observability, and sparse rewards, which exceed the difficulty of tasks that were previously solved by planning with learned models. PlaNet uses substantially fewer episodes and reaches final performance close to and sometimes higher than strong model-free algorithms. Model-free reinforcement learning (RL) can be used to learn effective policies for complex tasks, such as Atari games, even from image observations. However, this typically requires very large amounts of interaction -- substantially more, in fact, than a human would need to learn the same games. How can people learn so quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on a range of Atari games in low data regime of @math K interactions between the agent and the environment, which corresponds to two hours of real-time play. In most games SimPLe outperforms state-of-the-art model-free algorithms, in some games by over an order of magnitude. Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures. In model-based reinforcement learning, the agent interleaves between model learning and planning. These two components are inextricably intertwined. If the model is not able to provide sensible long-term prediction, the executed planner would exploit model flaws, which can yield catastrophic failures. This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration. To this end, we build a latent-variable autoregressive model by leveraging recent ideas in variational inference. We argue that forcing latent variables to carry future information through an auxiliary task substantially improves long-term predictions. Moreover, by planning in the latent space, the planner's solution is ensured to be within regions where the model is valid. An exploration strategy can be devised by searching for unlikely trajectories under the model. Our method achieves higher reward faster compared to baselines on a variety of tasks and environments in both the imitation learning and model-based reinforcement learning settings. We present an approach to sensorimotor control in immersive environments. Our approach utilizes a high-dimensional sensory stream and a lower-dimensional measurement stream. The cotemporal structure of these streams provides a rich supervisory signal, which enables training a sensorimotor control model by interacting with the environment. The model is trained using supervised learning techniques, but without extraneous supervision. It learns to act based on raw sensory input from a complex three-dimensional environment. The presented formulation enables learning without a fixed goal at training time, and pursuing dynamically changing goals at test time. We conduct extensive experiments in three-dimensional simulations based on the classical first-person game Doom. The results demonstrate that the presented approach outperforms sophisticated prior formulations, particularly on challenging tasks. The results also show that trained models successfully generalize across environments and goals. A model trained using the presented approach won the Full Deathmatch track of the Visual Doom AI Competition, which was held in previously unseen environments.