Learning competitive ensemble of information-constrained primitives
We want to develop reinforcement learning algorithms that enable the learning agent to obtain a structured decomposition of its behavior. Hierarchical Reinforcement Learning provides a mechanism for doing this by explicitly modularising the policy into two components --- a set of low-level sub-policies (or primitives) and a high-level master policy to coordinate between the primitives. While the primitives have to specialize to only a part of the state space, the master policy has to specialize to the entire state space as it decides when to activate which primitives. This introduces a        ``bottleneck'' where the success of the agent depends on the success of the master policy, thereby making it a single point of failure.  We propose to do away with this limitation by using a new mechanism where the sub-policies can decide for themselves in which part of the state they want to act. This decentralized decision making does away with the need for a parameterized master policy. We use this mechanism to train a policy that is composed of an ensemble of primitives but one that does not require a master policy to choose between the primitives. We experimentally demonstrate that this policy architecture improves over both flat and hierarchical policies in the terms of generalization. This work is under review at the NeurIPS 2019 Conference as a paper titled Learning Competitive Ensemble of Information-Constrained Primitives.  In Chapter One, I provide a background to Reinforcement Learning, Hierarchical Reinforcement Learning, Information Bottleneck, Compositionality, and Neural Module Networks and discuss how the proposed work in Chapter Two relates to these ideas. Chapter Two describes the idea of training an ensemble of primitives. I conclude the thesis by discussing some future research directions for the work described in Chapter Two.
Reinforcement Learning Hierarchical Reinforcement Learning Information Bottleneck Compositionality Modular network