In this paper, a new intrinsic reward generation method for sparse-reward reinforcement learning is proposed based on an ensemble of dynamics models. In the proposed method, the mixture of multiple dynamics models is used to approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the surprise seen from each dynamics model to the mixture of the dynamics models. In order to show the effectiveness of the proposed intrinsic reward generation method, a working algorithm is constructed by combining the proposed intrinsic reward generation method with the proximal policy optimization (PPO) algorithm. Numerical results show that for representative locomotion tasks, the proposed model-ensemble-based intrinsic reward generation method outperforms the previous methods based on a single dynamics model. Reinforcement learning (RL) with sparse reward is an active research area (Andrychowicz et al., 2017; de Abril & Kanai, 2018; Kim et al., 2018; Tang et al., 2017 ). In typical model-free RL, an agent learns a policy to maximize the expected cumulative reward under the circumstance that the agent receives a non-zero reward from the environment for each action of the agent. On the contrary, in sparse reward RL, the environment does not return a non-zero reward for every action of the agent but returns a non-zero reward only when certain conditions are met. Such situations are encountered in many action control problems (Andrychowicz et al., 2017; . As in conventional RL, exploration is important at the early stage of learning in sparse reward RL, whereas the balance between exploration and exploitation is required on the later stage. Methods such as the -greedy strategy (Mnih et al., 2015; Van Hasselt et al., 2016) and the control of policy gradient with Gaussian random noise Schulman et al., 2015a) have been applied to various tasks for exploration. However, these methods have been revealed to be insufficient for successful learning when reward is sparse (Achiam & Sastry, 2017) . In order to overcome such difficulty, intrinsically motivated RL has been studied to stimulate better exploration by generating intrinsic reward for each action by the agent itself, even when reward is sparse. Recently, many intrinsically-motivated RL algorithms have been devised to deal with the sparsity of reward, e.g., based on the notion of curiosity Pathak et al., 2017) and surprise (Achiam & Sastry, 2017) . It is shown that these algorithms are successful and outperform the previous approaches. In essence, these algorithms use a single estimation model for the next state or the environment dynamics to generate intrinsic reward. In this paper, in order to further improve the performance of sparse reward model-free RL, we propose a new method to generate intrinsic reward based on an ensemble of estimation models for the environment dynamics. The rationale behind our approach is that by using a mixture of several distributions, we can increase degrees of freedom for modeling the unknown underlying model dynamics and designing a better reward from the ensemble of estimation models. Numerical results show that the proposed model-ensemble-based intrinsic reward generation method yields improved performance as compared to existing reward generation methods for continuous control with sparse reward setting. In this paper, we have proposed a new intrinsic reward generation method based on an ensemble of dynamics models for sparse-reward reinforcement learning. In the proposed method, the mixture of multiple dynamics models is used to better approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the intrinsic reward computed from each dynamics model to the mixture to capture the most relevant surprise. The proposed intrinsic reward generation method was combined with PPO to construct a working algorithm. Ablation study has been performed to investigate the impact of the hyperparameters associated with the proposed ensemblebased intrinsic reward generation. Numerical results show that the proposed model-ensemble-based intrinsic reward generation method outperforms major existing intrinsic reward generation methods in the considered sparse environments. A PROOFS Proposition 1. Let η(π) be the actual expected discounted sum of extrinsic rewards defined in (8). Then, the following inequality holds: where c is a positive constant. Proof. The inequality (a) is trivial from the definition of π * , that is, π * is an optimal policy maximizing η(π). The inequality (b) holds since Proposition 2. Let P φ i (·|s, a), i = 1, . . . , K be the ensemble of model distributions, and P (·|s, a) be an arbitrary true transition probability distribution. Then, the minimum of average KLD between P (·|s, a) and the mixture model P = i q i P φ i (·|s, a) over the mixture weights {q 1 , · · · , q K |q i ≥ 0, i q i = 1} is upper bounded by the minimum of average KLD between P and P φ i over {i}: i.e., Proof. Here, (26) is valid due to the convexity of the KL divergence in terms of the second argument for a fixed first argument. (27) is valid due to the linearity of expectation. (28) is valid since the minimum in the right-hand side of (27) is achieved when we assign all the mass to q i that has the minimum value of E (s,a)∼ π * D KL P ||P φ i |(s, a) . (Note that the optimal {q i } in (27) is not the same as the optimal {q i } achieving the minimum in (25). ) Note that each step in the proof is tight except (26) in which the convexity of the KL divergence in terms of the second argument is used. This part involves the function f (x) = − log x for 0 < x ≤ 1 since D KL (p 1 ||p 2 ) = p 1 (y) log p1(y ) p2(y ) dy, but the convexity of f ( x) = − log x for 0 < x ≤ 1 is not so severe if x is not so close to zero.
For sparse-reward reinforcement learning, the ensemble of multiple dynamics models is used to generate intrinsic reward designed as the minimum of the surprise.
{"src": "scitldr-AIC_train", "paper_id": "SyxJU64twr"}