Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether modern deep reinforcement learning can be used to train agents to perform causal reasoning. We adopt a meta-learning approach, where the agent learns a policy for conducting experiments via causal interventions, in order to support a subsequent task which rewards making accurate causal inferences.We also found the agent could make sophisticated counterfactual predictions, as well as learn to draw causal inferences from purely observational data. Though powerful formalisms for causal reasoning have been developed, applying them in real-world domains can be difficult because fitting to large amounts of high dimensional data often requires making idealized assumptions. Our results suggest that causal reasoning in complex settings may benefit from powerful learning-based approaches. More generally, this work may offer new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform—and interpret—experiments. Many machine learning algorithms are rooted in discovering patterns of correlation in data. While this has been sufficient to excel in several areas BID20 BID7 , sometimes the problems we are interested in are fundamentally causal. Answering questions such as "Does smoking cause cancer?" or "Was this person denied a job due to racial discrimination?" or "Did this marketing campaign cause sales to go up?" all require an ability to reason about causes and effects and cannot be achieved by purely associative inference. Even for problems that are not obviously causal, like image classification, it has been suggested that some failure modes emerge from lack of causal understanding. Causal reasoning may be an essential component of natural intelligence and is present in human babies, rats and even birds BID23 BID14 BID15 BID5 BID21 . There is a rich literature on formal approaches for defining and performing causal reasoning BID29 BID33 BID8 BID30 ).Here we investigate whether procedures for learning and using causal structure can be produced by meta-learning. The approach of meta-learning is to learn the learning (or inference) procedure itself, directly from data. We adopt the specific method of BID9 and BID35 , training a recurrent neural network (RNN) through model-free reinforcement learning. We train on a large family of tasks, each underpinned by a different causal structure.The use of meta-learning avoids the need to manually implement explicit causal reasoning methods in an algorithm, offers advantages of scalability by amortizing computations, and allows automatic incorporation of complex prior knowledge BID1 BID35 BID11 . Additionally , by learning end-to-end, the algorithm has the potential to find the internal representations of causal structure best suited for the types of causal inference required. This work is the first demonstration that causal reasoning can arise out of model-free reinforcement learning. This opens up the possibility of leveraging powerful learning-based methods for causal inference in complex settings. Traditional formal approaches usually decouple the two problems of causal induction (i.e. inferring the structure of the underlying model) and causal inference (i.e. estimating causal effects and answering counterfactual questions), and despite advances in both BID26 BID6 BID27 BID32 BID12 BID22 , inducing models often requires assumptions that are difficult to fit to complex real-world conditions. By learning these end-to-end, our method can potentially find representations of causal structure best tuned to the specific causal inferences required. Another key advantage of our meta-RL approach is that it allows the agent to learn to interact with the environment in order to acquire necessary observations in the service of its task-i.e. to perform active learning. In our experimental domain, our agents' active intervention policy was close to optimal, which demonstrates the promise of agents that can learn to experiment on their environment and perform rich causal reasoning on the observations. Future work should explore agents that perform experiments to support structured exploration in RL, and optimal experiment design in complex domains where large numbers of blind interventions are prohibitive. To this end, follow-up work should focus on scaling up our approach to larger environments, with more complex causal structure and a more diverse range of tasks. Though the results here are a first step in this direction which use relatively standard deep RL components, our approach will likely benefit from more advanced architectures (e.g. BID16 BID17 that allow longer more complex episodes, as well as models which are more explicitly compositional (e.g. BID3 BID0 or have richer semantics (e.g. BID13 , that more explicitly leverage symmetries like equivalance classes in the environment. We can also compare the performance of these agents to two standard model-free RL baselines. The Q-total agent learns a Q-value for each action across all steps for all the episodes. The Q-episode agent learns a Q-value for each action conditioned on the input at each time step [o t ,a t−1 ,r t−1 ], but with no LSTM memory to store previous actions and observations. Since the relationship between action and reward is random between episodes, Q-total was equivalent to selecting actions randomly, resulting in a considerably negative reward. The Q-episode agent essentially makes sure to not choose the arm that is indicated by m t to be the external intervention (which is assured to be equal to −5), and essentially chooses randomly otherwise, giving an average reward of 0.
meta-learn a learning algorithm capable of causal reasoning
{"src": "scitldr-AIC_train", "paper_id": "H1ltQ3R9KQ"}