A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency. In a traditional reinforcement learning setting, an agent interacts with an environment in a sequence of episodes, observing states and acting according to a policy that ideally maximizes expected cumulative reward. If an agent is required to pursue different goals across episodes, its goal-conditional policy may be represented by a probability distribution over actions for every combination of state and goal. This distinction between states and goals is particularly useful when the probability of a state transition given an action is independent of the goal pursued by the agent.Learning such goal-conditional behavior has received significant attention in machine learning and robotics, especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent BID17 BID3 Kupcsik et al., 2013; Deisenroth et al., 2014; BID16 BID29 Kober et al., 2012; Ghosh et al., 2018; Mankowitz et al., 2018; BID11 . Consequently, developing goal-based curricula to facilitate learning has also attracted considerable interest (Fabisch & Metzen, 2014; Florensa et al., 2017; BID20 BID19 . In hierarchical reinforcement learning, goal-conditional policies may enable agents to plan using subgoals, which abstracts the details involved in lower-level decisions BID10 BID26 Kulkarni et al., 2016; Levy et al., 2017) .In a typical sparse-reward environment, an agent receives a non-zero reward only upon reaching a goal state. Besides being natural, this task formulation avoids the potentially difficult problem of reward shaping, which often biases the learning process towards suboptimal behavior BID9 . Unfortunately , sparse-reward environments remain particularly challenging for traditional reinforcement learning algorithms BID0 Florensa et al., 2017) . For example, consider an agent tasked with traveling between cities. In a sparse-reward formulation, if reaching a desired destination by chance is unlikely, a learning agent will rarely obtain reward signals. At the same time, it seems natural to expect that an agent will learn how to reach the cities it visited regardless of its desired destinations.In this context, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended is called hindsight. This capacity was recently introduced by BID0 to off-policy reinforcement learning algorithms that rely on experience replay (Lin, 1992) . In earlier work, Karkus et al. (2016) introduced hindsight to policy search based on Bayesian optimization BID5 .In this paper, we demonstrate how hindsight can be introduced to policy gradient methods BID27 BID28 BID22 , generalizing this idea to a successful class of reinforcement learning algorithms BID13 Duan et al., 2016) .In contrast to previous work on hindsight, our approach relies on importance sampling BID2 . In reinforcement learning , importance sampling has been traditionally employed in order to efficiently reuse information obtained by earlier policies during learning BID15 BID12 Jie & Abbeel, 2010; BID7 . In comparison, our approach attempts to efficiently learn about different goals using information obtained by the current policy for a specific goal. This approach leads to multiple formulations of a hindsight policy gradient that relate to well-known policy gradient results.In comparison to conventional (goal-conditional) policy gradient estimators, our proposed estimators lead to remarkable sample efficiency on a diverse selection of sparse-reward environments. We introduced techniques that enable learning goal-conditional policies using hindsight. In this context, hindsight refers to the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended. Prior to our work, hindsight has been limited to off-policy reinforcement learning algorithms that rely on experience replay BID0 and policy search based on Bayesian optimization (Karkus et al., 2016) .In addition to the fundamental hindsight policy gradient, our technical results include its baseline and advantage formulations. These results are based on a self-contained goal-conditional policy framework that is also introduced in this text. Besides the straightforward estimator built upon the per-decision hindsight policy gradient, we also presented a consistent estimator inspired by weighted importance sampling, together with the corresponding baseline formulation. A variant of this estimator leads to remarkable comparative sample efficiency on a diverse selection of sparsereward environments, especially in cases where direct reward signals are extremely difficult to obtain. This crucial feature allows natural task formulations that require just trivial reward shaping.The main drawback of hindsight policy gradient estimators appears to be their computational cost, which is directly related to the number of active goals in a batch. This issue may be mitigated by subsampling active goals, which generally leads to inconsistent estimators. Fortunately, our experiments suggest that this is a viable alternative. Note that the success of hindsight experience replay also depends on an active goal subsampling heuristic (Andrychowicz et al., 2017, Sec. 4.5) .The inconsistent hindsight policy gradient estimator with a value function baseline employed in our experiments sometimes leads to unstable learning, which is likely related to the difficulty of fitting such a value function without hindsight. This hypothesis is consistent with the fact that such instability is observed only in the most extreme examples of sparse-reward environments. Although our preliminary experiments in using hindsight to fit a value function baseline have been successful, this may be accomplished in several ways, and requires a careful study of its own. Further experiments are also required to evaluate hindsight on dense-reward environments.There are many possibilities for future work besides integrating hindsight policy gradients into systems that rely on goal-conditional policies: deriving additional estimators; implementing and evaluating hindsight (advantage) actor-critic methods; assessing whether hindsight policy gradients can successfully circumvent catastrophic forgetting during curriculum learning of goal-conditional policies; approximating the reward function to reduce required supervision; analysing the variance of the proposed estimators; studying the impact of active goal subsampling; and evaluating every technique on continuous action spaces. Theorem A.1. The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM0 Proof. The partial derivative ∂η(θ)/∂θ j of the expected return η(θ) with respect to θ j is given by DISPLAYFORM1 The likelihood-ratio trick allows rewriting the previous equation as DISPLAYFORM2 Note that DISPLAYFORM3 Therefore, DISPLAYFORM4 A.2 THEOREM 3.1Theorem 3.1 (Goal-conditional policy gradient). The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM5 Proof. Starting from Eq. 17, the partial derivative ∂η(θ)/∂θ j of η(θ) with respect to θ j is given by DISPLAYFORM6 The previous equation can be rewritten as DISPLAYFORM7 Let c denote an expectation inside Eq. 19 for t ≥ t. In that case, A t ⊥ ⊥ S t | S t , G, Θ, and so DISPLAYFORM8 Reversing the likelihood-ratio trick, DISPLAYFORM9 Therefore, the terms where t ≥ t can be dismissed from Eq. 19, leading to DISPLAYFORM10 The previous equation can be conveniently rewritten as DISPLAYFORM11 A.3 LEMMA A.1Lemma A.1. For every j, t, θ, and associated real-valued (baseline) function b DISPLAYFORM12 Proof. Letting c denote an expectation inside Eq. 24 , DISPLAYFORM13 Reversing the likelihood-ratio trick, DISPLAYFORM14 A.4 THEOREM 3.2 Theorem 3.2 (Goal-conditional policy gradient, baseline formulation). For every t, θ, and associated real-valued (baseline ) function b θ t , the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM15 Proof. The result is obtained by subtracting Eq. 24 from Eq. 23. Importantly, for every combination of θ and t, it would also be possible to have a distinct baseline function for each parameter in θ.A.5 LEMMA A.2 Lemma A.2. The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM16 Proof. Starting from Eq. 23 and rearranging terms, DISPLAYFORM17 By the definition of action-value function, DISPLAYFORM18 A.6 THEOREM 3.3Theorem 3.3 (Goal-conditional policy gradient, advantage formulation). The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM19 Proof. The result is obtained by choosing b θ t = V θ t and subtracting Eq. 24 from Eq. 29.A.7 THEOREM A.2For arbitrary j and θ, consider the following definitions of f and h. DISPLAYFORM20 DISPLAYFORM21 For every b j ∈ R, using Theorem 3.1 and the fact that DISPLAYFORM22 Proof. The result is an application of Lemma D.4. The following theorem relies on importance sampling, a traditional technique used to obtain estimates related to a random variable X ∼ p using samples from an arbitrary positive distribution q. This technique relies on the following equalities:
We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.
{"src": "scitldr-AIC_train", "paper_id": "Bkg2viA5FQ"}