The information bottleneck principle is an elegant and useful approach to representation learning. In this paper, we investigate the problem of representation learning in the context of reinforcement learning using the information bottleneck framework, aiming at improving the sample efficiency of the learning algorithms.We analytically derive the optimal conditional distribution of the representation, and provide a variational lower bound. Then, we maximize this lower bound with the Stein variational (SV) gradient method. 
 We incorporate this framework in the advantageous actor critic algorithm (A2C) and the proximal policy optimization algorithm (PPO). Our experimental results show that our framework can improve the sample efficiency of vanilla A2C and PPO significantly. Finally, we study the information-bottleneck (IB) perspective in deep RL with the algorithm called mutual information neural estimation(MINE).
 We experimentally verify that the information extraction-compression process also exists in deep RL and our framework is capable of accelerating this process. We also analyze the relationship between MINE and our method, through this relationship, we theoretically derive an algorithm to optimize our IB framework without constructing the lower bound. In training a reinforcement learning algorithm, an agent interacts with the environment, explores the (possibly unknown) state space, and learns a policy from the exploration sample data. In many cases, such samples are quite expensive to obtain (e.g., requires interactions with the physical environment). Hence, improving the sample efficiency of the learning algorithm is a key problem in RL and has been studied extensively in the literature. Popular techniques include experience reuse/replay, which leads to powerful off-policy algorithms (e.g., (Mnih et al., 2013; Silver et al., 2014; Van Hasselt et al., 2015; Nachum et al., 2018a; Espeholt et al., 2018 )), and model-based algorithms (e.g., (Hafner et al., 2018; Kaiser et al., 2019) ). Moreover, it is known that effective representations can greatly reduce the sample complexity in RL. This can be seen from the following motivating example: In the environment of a classical Atari game: Seaquest, it may take dozens of millions samples to converge to an optimal policy when the input states are raw images (more than 28,000 dimensions), while it takes less samples when the inputs are 128-dimension pre-defined RAM data (Sygnowski & Michalewski, 2016) . Clearly, the RAM data contain much less redundant information irrelevant to the learning process than the raw images. Thus, we argue that an efficient representation is extremely crucial to the sample efficiency. In this paper, we try to improve the sample efficiency in RL from the perspective of representation learning using the celebrated information bottleneck framework (Tishby et al., 2000) . In standard deep learning, the experiments in (Shwartz-Ziv & Tishby, 2017) show that during the training process, the neural network first "remembers" the inputs by increasing the mutual information between the inputs and the representation variables, then compresses the inputs to efficient representation related to the learning task by discarding redundant information from inputs (decreasing the mutual information between inputs and representation variables). We call this phenomena "information extraction-compression process" "information extraction-compression process" "information extraction-compression process"(information E-C process). Our experiments shows that, similar to the results shown in (Shwartz-Ziv & Tishby, 2017) , we first (to the best of our knowledge) observe the information extraction-compression phenomena in the context of deep RL (we need to use MINE (Belghazi et al., 2018) for estimating the mutual information). This observation motivates us to adopt the information bottleneck (IB) framework in reinforcement learning, in order to accelerate the extraction-compression process. The IB framework is intended to explicitly enforce RL agents to learn an efficient representation, hence improving the sample efficiency, by discarding irrelevant information from raw input data. Our technical contributions can be summarized as follows: 1. We observe that the "information extraction-compression process" also exists in the context of deep RL (using MINE (Belghazi et al., 2018) to estimate the mutual information). 2. We derive the optimization problem of our information bottleneck framework in RL. In order to solve the optimization problem, we construct a lower bound and use the Stein variational gradient method developed in (Liu et al., 2017) to optimize the lower bound. 3. We show that our framework can accelerate the information extraction-compression process. Our experimental results also show that combining actor-critic algorithms (such as A2C, PPO) with our framework is more sample-efficient than their original versions. 4. We analyze the relationship between our framework and MINE, through this relationship, we theoretically derive an algorithm to optimize our IB framework without constructing the lower bound. Finally, we note that our IB method is orthogonal to other methods for improving the sample efficiency, and it is an interesting future work to incorporate it in other off-policy and model-based algorithms. We study the information bottleneck principle in RL: We propose an optimization problem for learning the representation in RL based on the information-bottleneck framework and derive the optimal form of the target distribution. We construct a lower bound and utilize Stein Variational gradient method to optimize it. Finally, we verify that the information extraction and compression process also exists in deep RL, and our framework can accelerate this process. We also theoretically derive an algorithm based on MINE that can directly optimize our framework and we plan to study it experimentally in the future work. According to the assumption, naturally we have: Notice that if we use our IB framework in value-based algorithm, then the objective function J π can be defined as: where and d π is the discounted future state distribution, readers can find detailed definition of d π in the appendix of (Chen et al., 2018) . We can get:
Derive an information bottleneck framework in reinforcement learning and some simple relevant theories and tools.
{"src": "scitldr-AIC_train", "paper_id": "Syl-xpNtwS"}