Computer Science
Computer Science-2021.gz:143127
Computer Science
363ba1006cedaf56fa4b7a0073e847a18f6711ef
ADER:Adapting between Exploration and Robustness for Actor-Critic Methods

Combining off-policy reinforcement learning methods with function approximators such as neural networks has been found to lead to overestimation of the value function and sub-optimal solutions. Improvement such as TD3 has been proposed to address this issue. However, we surprisingly find that its performance lags behind the vanilla actor-critic methods (such as DDPG) in some primitive environments. In this paper, we show that the failure of some cases can be attributed to insufficient exploration. We reveal the culprit of insufficient exploration in TD3, and propose a novel algorithm toward this problem that ADapts between Exploration and Robustness, namely ADER. To enhance the exploration ability while eliminating the overestimation bias, we introduce a dynamic penalty term in value estimation calculated from estimated uncertainty, which takes into account different compositions of the uncertainty in different learning stages. Experiments in several challenging environments demonstrate the supremacy of the proposed method in continuous control tasks.


Introduction
Recent studies have revealed that combining off-policy reinforcement learning (RL) with function approximators (such as neural networks) can lead to systematic overestimation bias of value function (Hasselt 2010; Van Hasselt, Guez, and Silver 2016). To address this problem, several methods have been proposed to reduce the overestimation through introducing the supplementary value function for target value computation such as Double Q Learning (DDQN) (Hasselt 2010), and TD3 (Fujimoto, Van Hoof, and Meger 2018). The latter one is one of the most widely used actor-critic methods for its simplicity and effectiveness in continuous control problems, which takes the minimum estimated value given by two value functions as the target value. In spite of its success in many cases, we found that it lagged behind the vanilla actor-critic methods such as DDPG (Lillicrap et al. 2015), in both asymptotic performance and sampleefficiency, in some primitive environments.
In Figure 1(a) we show a toy case with a penguin at the bottom of the grid world trying to catch the fish at the top. A wall of fire blocks it, and the only way is through the storm grid in the center. The transitions in all the grids are clean and deterministic, except the storm grid that leads the Copyright © 2021, All authors are affiliated with Baidu,.Inc. penguin to a random neighboring grid that is irrelevant to its decision (For complete environment settings, please refer to Appendix ). On this grid environment, we evaluated the TD3 and DDPG algorithms. The experimental result in Figure 1(c) shows that TD3 performance falls behind DDPG. By inspecting the behaviors of the agents, we found that the agent trained with TD3 has a lower probability of reaching the global optimum compared with DDPG, which is to cross the storm to reach the target. We plot the visiting count of each grid during the whole training process in Figure 1( b). The figure shows that TD3 has a substantially lower visiting count to the storm grid than DDPG, which prevents the agent from reaching the goal.
Concluding from the above case, TD3 discourages some key explorations to certain extent, which has prevented it from achieving higher asymptotic performance. We argue that the minimum operation in TD3 conflicts with the principle of optimism in the face of uncertainty, which is related to uncertainty of estimation (Lai and Robbins 1985;Strehl and Littman 2005). This means that TD3 reduces the overestimation bias of the value function at the cost of insufficient exploration, which can lead to sub-optimal policies. We will present the full analysis in Section .
In this paper, we propose a novel algorithm called ADER, considering the estimation uncertainty in value estimation. Estimation uncertainty has mainly two components (Dabney et al. 2018;Mavrin et al. 2019): parametric uncertainty, which arises from prediction variance of function approximators given finite training samples, and intrinsic uncertainty, which arises from the inherent stochasticity of the environment. The parametric uncertainty decreases as more training samples are acquired, while the intrinsic uncertainty persists along with the training. Both kinds of uncertainty may lead to estimation bias, but parametric uncertainty acts a more important role for exploration. Although it is difficult to distinguish the different components precisely, we can introduce an automatic tuning schedule to emphasize different uncertainties at different learning stages: at the early learning stage when data is insufficient, minor penalty on parametric uncertainty can facilitate exploration, and at the later training stage when the intrinsic uncertainty dominates estimation uncertainty, high penalty on intrinsic uncertainty can reduce the overestimation bias. In this way, we are able to retain efficient exploration and reduce overestimation bias at arXiv:2109.03443v1 [cs.
LG] 8 Sep 2021 the same time. We evaluated the algorithm in several challenging continuous control environments. Experimental results show that our algorithm benefits both exploration efficiency and policy robustness, achieving better performance than the existing RL methods.

Related Work
Exploration with optimism in the face of uncertainty has been widely adopted in deep RL methods. The count-based approach (Tang et al. 2017) estimates the uncertainty with the state visitation counts, which are then used as the reward bonus. Curiosity-driven exploration (Pathak et al. 2017;Badia et al. 2020) introduces a reward bonus for exploration that is related to how much the agent knows the environment. Bootstrapped DQN (Osband et al. 2016) employs a multi-headed neural network to represent the ensemble Qvalues to perform exploration. The variance of the estimated values given by different heads can also be used as a metric of the estimation uncertainty. Among these methods, the estimation uncertainty is employed as a reward bonus, but our investigation on TD3 shows that its update is taking the estimation uncertainty as a penalty for reward shaping. There are two sources of uncertainty associated with exploration in model estimation.: parametric uncertainty and intrinsic uncertainty (Lai and Robbins 1985;Dabney et al. 2018;Mavrin et al. 2019). While the parametric uncertainty decreases as the size of training data increases, the intrinsic uncertainty persists along with the training as it arises from the inherent systematic stochasticity of the environment. There are some methods that (Moerland, Broekens, and Jonker 2017) seek to combine both types of uncertainty to improve exploration, with Bayesian drop-out (Gal and Ghahramani 2016) for parametric uncertainty estimation, and Gaussian distribution for intrinsic uncertainty estimation. While it is hard to quantify different types of uncertainty, the distributional RL method (Mavrin et al. 2019) suppresses the intrinsic uncertainty by introducing a decaying schedule for robust policy learning, avoiding selecting the actions with higher variance.
There have also been other works of exploration improvement over actor-critic methods. Soft Actor-Critic (SAC) (Haarnoja et al. 2018) performs policy learning with entropy maximization, which can acquire diverse behaviors and improve exploration. Optimistic Actor-Critic (OAC) (Ciosek et al. 2019) collects the training data with a behavior policy derived from the training policy for optimistic exploration and performs the off-policy update. Following the principle of optimism in the face of uncertainty. The behavior policy was obtained by maximizing the approximate of the upper confidence bound of the state-action value estimation. The main difference between OAC and our work is that we improve the exploration efficiency of the training policy, while OAC optimizes the behavior policy.

Actor-Critic Methods
In Reinforcement learning(RL), an agent learns to interact with the environment (Sutton and Barto 2018) following Markov Decision Processes (MDP). We use s ∈ S to denote the state, and a ∈ A to denote the action, which depends on the policy π(a|s). The environment returns a reward r, and informs the next state s following the transition p(s |a, s), which is normally unknown to the agent. The goal of the problem is to find the optimal policy π * that maximizes the expected future reward: The state-action value function under policy π is defined as Q In Qlearning (Watkins and Dayan 1992), the state-action value function Q is updated by minimizing the temporal difference (TD) error: DQN (Mnih et al. 2015) further uses the neural network as the function approximator in Q-learning algorithm. The authors proposed two improvements to enhance the convergence stability: experience replay buffer and target network. The transitions (s, a, r, s ) are stored in a first-in-first-out (FIFO) queue named experience replay buffer, and samples will be sampled uniformly and repeatedly for training. The target network Q θ − maintains a delayed copied parameters of the training network Q θ . The target values are computed with the target network: For continuous control problems, it is necessary to introduce a policy network π φ (a|s), which formed the actor-critic method. For instance, the deep deterministic policy gradient (DDPG) (Lillicrap et al. 2015) updates the policy network by maximizing the state-action value function as follow, where the state-action pair (s, a) is uniformly sampled from the experience replay buffer. For stable training with neural networks, DDPG also maintains a pair of target networks for actor and critic networks (Q θ − , π φ − ) that slowly track the training parameters:

Overestimation Bias
The maximum operator of Equation 2 lead to systematic overestimation bias, due to the unavoidable prediction error occurs in estimated values, considering the following fact (Thrun and Schwartz 1993): To address this problem, Double DQN (Van Hasselt, Guez, and Silver 2016) uses two value functions to disentangle action selection from value estimation, one for value estimation, and another for action selection. The overestimation bias persists in actor-critic methods such as DDPG (Fujimoto, Van Hoof, and Meger 2018). The TD3 algorithm employs two distinct neural networks (Q θ1 , Q θ2 ) for value estimation, taking the minimum between the two for calculating the target value:

The Issue of Insufficient Exploration
With a few calculations, the target value in Equation 6 can be represented in the following form: where µ and σ represent the statistical mean and standard variance respectively. We provide the derivation in Appendix . Equation 7 shows that the minimum operation in TD3 is equal to approximating the lower confidence bound of the value estimation by subtracting the uncertainty from the mean value.
On the other hand, effective exploration has been one of the main challenges in deep reinforcement learning. Recent work on uncertainty-based exploration shows that exploration guided by estimation uncertainty can enhance the exploration efficiency (Bellemare et al. 2016), which employs the following target: where N (s, a) denotes the visiting count of the state action pair (s, a). Previous theory has revealed that the parametric uncertainty of any estimation at some point s is proportional to the reciprocal of the square root of the visiting count (Auer, Cesa-Bianchi, and Fischer 2002;Koenker 2005), and considering the visiting count of (s, a) is proportional to that of the state s by p(s |s, a), the second term in the right-hand side of Equation 8 is also proportional to the parametric uncertainty of the value function at s . This is also called the principle of optimism in the face of uncertainty.
As the stand derivation of the estimates given by ensemble models are often used as the metric of estimating uncertainty (Smith 2001;Osband et al. 2016;Lakshminarayanan, Pritzel, and Blundell 2017), comparing Equation 7 and Equation 8 it is not hard to find out why TD3 lead to the insufficient exploration problem. While TD3 uses estimation uncertainty as a punishment in the target value to address the overestimation bias, it discourages exploring the states with high parametric uncertainty. Thus, a key contribution of our work is to solve this conflict to achieve better performance.

Methodology
Generally, estimation uncertainty is composed of the parametric and intrinsic uncertainty, and it is difficult to distinguish them from each other. However, these two sources of uncertainty have different trends in training. We thus introduce an automatic schedule to dynamically adjust the penalty on estimation uncertainty, which can benefit the exploration at the early stage of training and reduce the overestimation bias at the later stage. We describe the algorithm for ADER in Section 4.1, and justify it in Section 4.2.

Formulation
For convenience we are considering the value update for transition s, a, s . We use µ, σ to denote the mean and uncertainty of (Q 1 , Q 2 ) respectively. We further use σ I and σ P to denote the intrinsic and parametric uncertainty respectively. Note that we have σ = σ I + σ P . Following the aforementioned analysis, the ideal computation of target values can be written as: where α > 0 determines the penalty on intrinsic uncertainty to reduce the overestimation bias, and β > 0 indicates the bonus on parametric uncertainty for exploration. This allows the algorithm to explore the state space following the parametric uncertainty and to reduce the overestimation bias introduced by the intrinsic uncertainty at the same time. Unfortunately, we can not precisely estimate σ P and σ I separately. However, it is possible to approximate σ through an ensemble of neural networks. Formally, we define the ADER algorithm as using the following target value: where α and κ are constant values, t is an epoch counter starting from t = 2 and updated every K environment steps. By setting α = 1 and κ = 0, ADER degenerates to TD3. With α = 0 and κ = 0, ADER degenerates to the DDPG algorithm with two ensemble Q-value functions.

Interpreting ADER
Starting from the target value computation in TD3, for an arbitrary state-action pair, we wish to find the η(t) to adjust the penalty on estimation uncertainty such that the target values computed by Equation 10 is identical to the values computed by Equation 9: Then the resulting η(t) has the following approximate form: The proof is deferred to Appendix . Since σ P rises from the environment property and remains constant, the right term of Equation 12 is proportional to σ P : (α + β) σ P σ I ∝ σ P . Given that the decay of parametric uncertainty is statistically proportional to the rate: σ P ∝ c logt t (Koenker 2005;Mavrin et al. 2019), where c is a constant value, we approximately replace the right term in Equation 12 with c logt t , and we can get: η(t) ≈ α − κ logt t , where κ = (α + β)c. Thus ADER introduces two additional hyper-parameters: α and κ. Note that α determines the penalty on intrinsic uncertainty in the later training stage: lim t→∞ η(t) = α. A higher value of α can benefit the policy robustness, since the penalty on intrinsic uncertainty encourages the deployment policy to be less likely to select the actions with high variance in future return. Once we set the value of α, κ indicates how fast the bonus of parametric uncertainty for exploration decays.

Training Stability
Reinforcement learning methods have been known to be unstable when combined with nonlinear function approximators (such as neural networks), and it is essential to maintain the target networks to stabilize the training (Mnih et al. 2015;Lillicrap et al. 2015). As the value of α depends on the environment steps, it changes rapidly and makes the distribution of target values unstable. We follow the update of target networks in DQN (Mnih et al. 2015) and update the α periodically. That is, we update the value of α every M train steps, rather than update it instantly after each epoch. We provide a full algorithm description in Algorithm 1.

Algorithm 1 ADER
Require: Initialize actor network : π φ , and 2 critic networks: Collect transitions {s, a, r, s , a } using the policy π φ , and append the data into B

Experiments
We evaluate ADER on Mujoco environments (Todorov, Erez, and Tassa 2012), and Roboschool (Klimov and Schulman 2017) environments (Figure 2). The baseline algorithms are state-of-the-art RL algorithms in the continuous control domain, including PPO , TD3, and SAC (Haarnoja et al. 2018). First, we compare the proposed algorithm against baselines in 8 challenging tasks. Then we perform ablations on each one of the improvements we propose in ADER. We also provide an additional analysis showing that ADER can enhance policy robustness and exploration. Each experiment was run four times with different random seeds.

Implementation and Results
Our implementation of ADER is based on the TD3 implementation provided by the authors. We keep all the hyperparameters the same as TD3, except for the additional hyperparameters introduced by ADER. The selection of additional hyper-parameters is obtained by the grid search method (Larochelle et al. 2007), with search range of α being [0, 3] with step 0.3, and the search range of κ being [1, 10] with step 1.0. For an exhaustive description of the hyperparameters, please refer to Appendix . For implementations of the baseline algorithms, we used the code provided by the authors. Note that for fair comparison in sample efficiency, following the experiment setting in TD3, we train exactly 1 iteration step for all the off-policy algorithms, while the authors of SAC reported the experimental results with 4 iteration step. To study how the exploration efficiency affects the asymptotic performance, we run the experiment in each environment with 5 million steps, far more than the 1 million steps used in TD3 experiments. We present the experimental results in Figure 3. ADER achieves the best average performance except for Swimmer-v1 and Humanoid-v1. For most environments, ADER falls behind TD3 in the first 1 million steps, but it soon surpasses the performance of TD3. It shows that ADER has paid some cost at the early training stage to encourage more exploration, but this could indeed help to achieve better performance in the subsequent stage. It is also worth noticing that both TD3 and ADER are successful in quite challenging en-vironments such as Humanoid-v1, RoboschoolHumanoid-v1, and RoboschoolHumanoidFlagrun-v1, where the control dimension is more challange, and the agent is prone to fall over. In contrast, without addressing the approximation error, DDPG has the poorest performance in those environments. Thus the main performance gain over DDPG can probably be attributed to the improvement of policy robustness by addressing the approximation error. Also, on-policy PPO has shown poor sample efficiency and thus lags behind the other off-policy algorithms in many cases.

Ablation Study
To understand how each specific component affects the performance, we perform ablative experiments and compare the following variants of ADER: • TD3: We remove the penalty adaption schedule and set α = 1, κ = 0 in ADER, which degenerates to TD3.
• No-PU: The basic setting of ADER without Periodical Update of η(t).
• HPS or Hyper-Parameter Search. ADER with best combination of α and κ using the grid search method. We present the ablation results in Table 1. The largest performance gain comes from the step from TD3 to Basic, Table 1: Average return of ADER variants on tested environments. The maximum value for each task is bolded, except the HPS variant. Most of the performance gain comes from dynamically adjusting the penalty on estimation uncertainty, and the performance can be further improved by searching a proper combination of hyper-parameters of α and κ.  Figure 4: The policy learned with higher α is more robust than those learned with lower α. Each learned policy was evaluated with 10 episodes at each point. The fall over rate indicates how many times the agent runs steadily without falling over in 10 episodes.
which demonstrates the importance of dynamic adjustments through η(t). Periodically updating the α is also essential for stabilizing the training process. The No-PU group falls behind the basic group in nearly all the tested environments except in RoboshoolHumanoid-v1, which is consistent with the ablative experiments on the target network reported in DQN (Mnih et al. 2015). After setting a smaller value for α (No-RI), we observe the performance drop in five environments such as HalfCheetah-v1 and Hopper-v1. We argue that a small α will lead the policy to favor with risky actions and collapse the performance, while a large α discourages the exploration. We will further discuss on the selection of α in the following part. The last column obtaining superior performance shows that the performance can be further improved by searching the combination of two hyper-parameters of α and κ specifically for each environment. One possible reason is that the decay rate of parametric uncertainty varies from environment to environment, due to the different environmental complexity.
To further investigate how increasing α helps the robustness, we studied more variants of ADER with α given different values in the Walker2d-v1 environment. Rather than use environmental score as a metric, we plot the fall over rate against the environment steps, resulting in Figure 4. It turns out that the fall over rate decreases steadily as we increase α before α < 2.5, but further increasing α to 3.0 becomes Figure 5: The visitation counts of the states collected in the training of the Walker2d-v1 environment. We project all the states into a 2d plane to visualize the state distribution.
counter-productive as exploration is severely suppressed.

Exploration Efficiency
To validate that ADER facilitates the exploration, we study the diversity of visited states during the training process. We collect the 5 million states during the experiments reported in Section and projected them into a 2d plane to visualize the state distribution ( Figure 5), using the principal component analysis (PCA) to reduce the state dimension (Jolliffe and Cadima 2016). In the figure of ADER, the bright points are distributed evenly in the plane, while most of the points of TD3 are concentrated in the center. It indicates that ADER can lead to a richer set of states and improve exploration.

Conclusion
In this work, we research the insufficient exploration problem of TD3, which can lead to locally optimal solutions. Inspired by the principle of optimism in the face of uncertainty, we present the ADER algorithm as an improvement. ADER uses an automatic schedule to deal with two sources of estimation uncertainty, which encourages exploration at the early training stage and reduces the overestimation bias at the later stage. Empirically, we show that ADER achieves better performance than state-of-the-art RL methods while benefiting the exploration and policy robustness. A possible extension of this work could be state-dependent tuning of the penalty factors. Also, as uncertainty estimation is important for both exploration and policy robustness, it is of interest to precisely model the parametric and intrinsic uncertainty separately.