Cooperation and communication in multiagent deep reinforcement learning
Acknowledgments Many thanks to my advisor Peter Stone for many years of support and encouragement throughout the PhD process. In particular, I remember a dark time when I was directionless and depressed about the prospect of finding fruitful research directions or ever graduating. You told me that all PhDs go through a period where they are lost in the woods and encouraged me to continue onward. This thesis is testament to the fact that given sufficient persistence, encouragement, and caffeine, there is a way out of the woods. Thanks to my lab mates and colleagues who have shaped my research and added flavor to my graduate experience: for the time and feedback they so generously gave. Thanks to Kay Nettle and Amy Bush for tirelessly supporting and debug ging the many issues that arise from setting up and maintaining a GPU cluster. The research in this thesis would have been impossible (or at least computationally intractable) without your help. Special thanks to my loving family and my fiance Man Liang for their unconditional support throughout the ups and downs of the PhD process. Reinforcement learning is the area of machine learning concerned with learning which actions to execute in an unknown environment in order to maximize cumulative reward. As agents begin to perform tasks of genuine interest to humans, they will be faced with environments too complex for humans to predetermine the correct actions using hand designed solutions. Instead, capable learning agents will be necessary to tackle complex real world domains. However, traditional reinforcement learning algorithms have difficulty with domains featuring 1) high dimensional continuous state spaces, for example pixels from a camera image, 2) high dimensional parameterized continuous action spaces, 3) partial observabil ity, and 4) multiple independent learning agents. We hypothesize that deep neural networks hold the key to scaling reinforcement learning towards complex tasks. This thesis seeks to answer the following two part question: 1) How can the power of Deep Neural Networks be leveraged to extend Reinforcement Learning to complex environments featuring partial observability, high dimensional parameterized continuous state and action spaces, and sparse re wards? 2) How can multiple Deep Reinforcement Learning agents learn to cooperate in a multiagent setting? To address the first part of this question, this thesis explores the idea of using recurrent neural networks to combat partial observability experienced by agents in the domain of Atari 2600 video games. Next, we design a deep reinforcement learning agent capable of discovering effective policies for