In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observation. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don't model, and can therefore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, but avoid the need to fully model future observations. The ability to predict future outcomes of hypothetical decisions is a key aspect of intelligence. One approach to capture this ability is via model-based reinforcement learning (MBRL) (Munro, 1987; Werbos, 1987; Nguyen & Widrow, 1990; Schmidhuber, 1991) . In this framework, an agent builds an internal representation s t by sensing an environment through observational data y t (such as rewards, visual inputs, proprioceptive information) and interacts with the environment by taking actions a t according to a policy π(a t |s t ). The sensory data collected is used to build a model that typically predicts future observations y >t from past actions a ≤t and past observations y ≤t . The resulting model may be used in various ways, e.g. for planning (Oh et al., 2015; Silver et al., 2017a) , generation of synthetic training data (Weber et al., 2017) , better credit assignment (Heess et al., 2015) , learning useful internal representations and belief states (Gregor et al., 2019; Guo et al., 2018) , or exploration via quantification of uncertainty or information gain (Pathak et al., 2017) . Within MBRL, commonly explored methods include action-conditional, next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et al., 2018; Kaiser et al., 2019; Janner et al., 2019) . However, it is often not tractable to accurately model all the available information. This is both due to the fact that conditioning on high-dimensional data such as images would require modeling and generating images in order to plan over several timesteps (Finn & Levine, 2017) , and to the fact that modeling images is challenging and may unnecessarily focus on visual details which are not relevant for acting. These challenges have motivated researchers to consider simpler models, henceforth referred to as partial models, i.e. models which are neither conditioned on, nor generate the full set of observed data (Guo et al., 2018; Gregor et al., 2019; Amos et al., 2018) . In this paper, we demonstrate that partial models will often fail to make correct predictions under a new policy, and link this failure to a problem in causal reasoning. Prior to this work, there has been a growing interest in combining causal inference with RL research in the directions of non-model based bandit algorithms (Bareinboim et al., 2015; Forney et al., 2017; Zhang & Bareinboim, 2017; Lee & Bareinboim, 2018; Bradtke & Barto, 1996) and causal discovery with RL (Zhu & Chen, 2019) . Contrary to previous works, in this paper we focus on model-based approaches and propose a novel framework for learning better partial models. A key insight of our methodology is the fact that any piece of information about the state of the environment that is used by the policy to make a decision, but is not available to the model, acts as a confounding variable for that model. As a result, the learned model is causally incorrect. Using such a model to reason may lead to the wrong conclusions about the optimal course of action as we demonstrate in this paper. We address these issues of partial models by combining general principles of causal reasoning, probabilistic modeling and deep learning. Our contributions are as follows. • We identify and clarify a fundamental problem of partial models from a causal-reasoning perspective and illustrate it using simple, intuitive Markov Decision Processes (MDPs) (Section 2). • In order to tackle these shortcomings we examine the following question: What is the minimal information that we have to condition a partial model on such that it will be causally correct with respect to changes in the policy? (Section 4) • We answer this question by proposing a family of viable solutions and empirically investigate their effects on models learned in illustrative environments (simple MDPs and 3D environments). Our method is described in Section 4 and the experiments are in Section 5. We have characterized and explained some of the issues of partial models in terms of causal reasoning. We proposed a simple, yet effective, modification to partial models so that they can still make correct predictions under changes in the behavior policy, which we validated theoretically and experimentally. The proposed modifications address the correctness of the model against policy changes, but don't address the correctness/robustness against other types of intervention in the environment. We will explore these aspects in future work.
Causally correct partial models do not have to generate the whole observation to remain causally correct in stochastic environments.
{"src": "scitldr-AIC_train", "paper_id": "HyeG9yHKPr"}