Deep reinforcement learning algorithms have proven successful in a variety of domains. However, tasks with sparse rewards remain challenging when the state space is large. Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the final goal is accomplished. In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum. This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning. We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario Bros and show that our method performs more efficiently and robustly than prior approaches in tasks with long time horizons and large state space. In addition, we demonstrate that using an optional keyframe scheme with very small quantity of key states, our approach can solve difficult robot manipulation challenges directly from perception and sparse rewards. Reinforcement learning (RL) aims to learn the optimal policy of a certain task by maximizing the cumulative reward acquired from the environment. Recently, deep reinforcement learning has enjoyed great successes in many domains with short-term and dense reward feedback BID16 ) (e.g. Atari games, TORCS). However, many real world problems are inherently based on sparse, binary rewards, where the agent needs to travel through a large number of states before a success (or failure) signal can be received. For instance, in the grasping task of a robotic arm, the most accurate task reward would be a binary reward only if the agent successfully picked up the the object or not. This kind of goal-oriented tasks with sparse rewards are considered the most difficult challenges in reinforcement learning. That is, the environment only provides a final success signal when the agent has accomplished the whole task. The search space of these tasks may exponentially expand as state chain extends, which adds to the difficulty of reaching the final goal with conventional reinforcement learning algorithms.There have been multiple lines of approaches for tackling such sparse reward problems, including the ideas based on intrinsic motivation BID23 BID3 BID10 , hierarchical reinforcement learning BID6 BID13 , curriculum learning BID12 BID24 and experience-based off-policy learning BID2 . Recently, a particularly promising approach is the idea of reverse curriculum generation BID7 . By reversing the traditional reinforcement learning process and gradually expanding the set of start states from the goal to the starting point, this method does achieve substantial progress in some simulation tasks. However, the newly sampled start set mixes old and new states, which leads to inefficiency in training, as the agent spends a large amount of time reviewing rather than learning new skills. This problem aggravates in large state space tasks where the expansion could be really slow, and the storing of history data is also impracticable due to the memory limitation.To solve goal-oriented tasks and avoid the drawbacks mentioned above, we propose a Tendency Reinforcement Learning (TRL) architecture which shapes a reward function with former experience and use it to stabilize and accelerate training. To achieve this, a discriminative reward (the "tendency") is shaped to output a judgment on the success tendency of each state, and the reward function for the agent is hybrid, which combines the final goal reward with the tendency hints. We define a set of start states as a phase, and continue extending it from the final goal, until it reaches where the task starts. In each phase, the tendency reward influences the agent's training. After training each phase, the tendency reward is updated using collected experience. This mutual promotion contributes to a dramatic improvement in training efficiency and no longer requires keeping all history data.In this paper, we introduce three novel techniques: First, we propose a hybrid reward design that combines the final reward with tendency hints. Secondly, with the introduction of a phase administrator, the curriculum for each phase is automated, which increases the efficiency. Lastly, we develop an optional keyframe scheme, that is, our framework is also compatible with additional keyframes with no strict accuracy requirement, as long as they are effective at reducing irrelevant search space.The major contribution of this work is that we present a reliable tendency reinforcement learning method that is capable of training agents to solve large state space tasks with only the final reward, taking raw pixels as perception. In our experiments, we apply and test the model in diverse domains, including a maze game, Super Mario Bros and 3 simulated tasks (a robotic grasping task, a crane conveyance task and a pick-and-place challenge), where it proves more efficient and competitive than the prior works. We develop a tendency reinforcement learning algorithm that resolves complicated goal-oriented tasks, and evaluate it with multiple experiments based on raw perceptions. Our method consists of two core components: a binary tendency classifier that provides dense hints on whether a state leads to success and a phase administrator that generates a reverse curriculum. In five distinct environments, we find that tendency RL method is efficient and stable enough to converge on a reliable policy within hours even facing intractable problems, and does not require history data which is impossible to acquire in large space domains.One limitation of our current approach is that our model requires the environment to be capable of resetting to some arbitrary states, which is also a limitation of the reverse curriculum generation method BID7 . A promising future direction is to additionally train reset controllers that automatically reset the system to the initial state following each episode, together with a corresponding forward controller BID8 .In future work, our goal is to apply the tendency RL method to real world robotic manipulation tasks with perceptions. Furthermore , we plan to extend the binary tendency classifier into a predictor that outputs the expected success rate on current state, so that our model can also be applied to stochastic environments. Additionally , we intend to release a subset of our code and the trained models to facilitate further research and comparisons.Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. A DEFINITIONS AND FORMULAE The regular reverse curriculum algorithm usually fails if there exists an irreversible process in the system. The irreversible process is defined as:∃s, s ∈ S : (∃n > 0 : P (s n = s |s 0 = s) > 0) ∧ (∀n > 0 : P (s n = s|s 0 = s ) = 0) (6)In such cases, the states s and s are not connected, and an agent starting from s will never reach s since that probability is 0. We define a absorbing state s a as a state that satisfies P (s a |s a ) = 1 ∧ ∀s ∈ S : s = s a −→ P (a|s) = 0To be more generalized, we define a set S a of states to be a absorbing set if it satisfies P (s |s) = 0 if s ∈ S a ∧ s / ∈ S aConsider a phase extension progress where P i+1 is generated from P i , if a large portion of states in P i+1 belong to some absorbing sets, it would be hard to for the new phase to include elements not in these absorbing sets. Therefore, the training is likely to be contained within these sets and no actual progress could be made since the phase extension makes no sense.However, with additional keyframes, this limitation could be avoided even with irreversible processes and absorbing sets. The mechanism is described as follows: When we sample states nearby a keyframe that is not in any absorbing set, the sampled states might happen to belong to some absorbing set. Although phase extensions are constrained within the absorbing sets, the generated phase might also cover some states sampled nearby a keyframe. Thus, according to the phase administrator algorithm (Alg. 2), that keyframe could still be reached.
We propose Tendency RL to efficiently solve goal-oriented tasks with large state space using automated curriculum learning and discriminative shaping reward, which has the potential to tackle robot manipulation tasks with perception.
{"src": "scitldr-AIC_train", "paper_id": "rJssAZ-0-"}