Many practical robot locomotion tasks require agents to use control policies that can be parameterized by goals. Popular deep reinforcement learning approaches in this direction involve learning goal-conditioned policies or value functions, or Inverse Dynamics Models (IDMs). IDMs map an agent’s current state and desired goal to the required actions. We show that the key to achieving good performance with IDMs lies in learning the information shared between equivalent experiences, so that they can be generalized to unseen scenarios. We design a training process that guides the learning of latent representations to encode this shared information. Using a limited number of environment interactions, our agent is able to efficiently navigate to arbitrary points in the goal space. We demonstrate the effectiveness of our approach in high-dimensional locomotion environments such as the Mujoco Ant, PyBullet Humanoid, and PyBullet Minitaur. We provide quantitative and qualitative results to show that our method clearly outperforms competing baseline approaches. In reinforcement learning (RL), an agent optimizes its behaviour to maximize a specific reward function that encodes tasks such as moving forward or reaching a target. After training, the agent simply executes the learned policy from its initial state until termination. In practical settings in robotics, however, control policies are invoked at the lowest level of a larger system by higher-level components such as perception and planning units. In such systems, agents have to follow a dynamic sequence of intermediate waypoints, instead of following a single policy until the goal is achieved. A typical approach to achieving goal-directed motion using RL involves learning goal-conditioned policies or value functions (Schaul et al. (2015) ). The key idea is to learn a function conditioned on a combination of the state and goal by sampling goals during the training process. However, this approach requires a large number of training samples, and does not leverage waypoints provided by efficient planning algorithms. Thus, it is desirable to learn models that can compute actions to transition effectively between waypoints. A popular class of such models is called Inverse Dynamics Model (IDM) (Christiano et al. (2016) ; Pathak et al. (2017) ). IDMs typically map the current state (or a history of states and actions) and the goal state, to the action. In this paper, we address the need of an efficient control module by learning a generalized IDM that can achieve goal-direction motion by leveraging data collected while training a state-of-the-art RL algorithm. We do not require full information of the goal state, or a history of previous states to learn the IDM. We learn on a reduced goal space, such as 3-D positions to which the agent must learn to navigate. Thus, given just the intermediate 3-D positions, or waypoints, our agent can navigate to the goal, without requiring any additional information about the intermediate states. The basic framework of the IDM is shown in Fig. 1 . The unique aspect of our algorithm is that we eliminate the need to randomly sample goals during training. Instead, we exploit the known symmetries/equivalences of the system (as is common in many robotics settings) to guide the collection of generalized experiences during training. We propose a class of algorithms that utilize the property of equivalence between transitions modulo the difference in a fixed set of attributes. In the locomotion setting, the agent's transitions are symmetric under translations and rotations. We capture this symmetry by defining equivalence modulo orientation among experiences. We use this notion of equivalence to guide the training of latent representations shared by these experiences and provide them as input to the IDM to produce the desired actions, as shown in Fig. 4 . A common challenge faced by agents trained using RL techniques is lack of generalization capability. The standard way of training produces policies that work very well on the states encountered by the agent during training, but often fail on unseen states. Achieving good performance using IDMs requires both these components: collecting generalized experiences, and learning these latent representations, as we demonstrate in Section 6. Our model exhibits high sample efficiency and superior performance, in comparison to other methods involving sampling goals during training. We demonstrate the effectiveness of our approach in the Mujoco Ant environment (Todorov et al. (2012) ) in OpenAI Gym (Brockman et al. (2016) ), and the Minitaur and Humanoid environments in PyBullet (Coumans & Bai (2016) ). From a limited number of experiences collected during training under a single reward function of going in one direction, our generalized IDM succeeds at navigating to arbitrary goal positions in the 3-D space. We measure performance by calculating the closest distance to the goal an agent achieves. We perform ablation experiments to show that (1) collecting generalized experience in the form of equivalent input pairs boosts performance over all baselines, (2) these equivalent input pairs can be condensed into a latent representation that encodes relevant information, and (3) learning this latent representation is in fact critical to success of our algorithm. Details of experiments and analysis of results can be found in Sections 5 and 6. We propose a new algorithm to achieve goal-directed motion for a variety of locomotion agents by learning the inverse dynamics model on shared latent representations for equivalent experiences. To this end, we take three important steps: (1) We utilize the experience collected by our agent while training a standard reinforcement learning algorithm, so that our IDM has "good" samples in which the agent walks reasonably well. (2) We generalize this experience by modifying the initial configuration for each observed trajectory in the collected data, and generate the equivalent trajectories. (3) We learn the important shared information between such symmetric pairs of experience samples through a latent representation that is used as an input to the IDM to produce the action required for the agent to reach the goal. We provide extensive qualitative and quantitative evidence to show that our methods surpass existing methods to achieve generalization over unseen parts of the state space. (Brockman et al., 2016) , and Humanoid and Minitaur environments in PyBullet (Coumans & Bai, 2016) . In each of these environments, for our methods, the agent is trained to perform well on the 3-D locomotion task in one direction. For the baseline methods, the agent is trained to reach goals generated in different 3-D positions. The reward function for collecting data for RL, GE and LR includes contact and control costs, and rewards for moving forward. The reward function for training VGCP includes contact and control costs, but instead of moving forward, the agent is encouraged to move in the direction of the goal. In the case of HER-Sparse, the agent only receives a reward 0 if it reaches the goal, and -1 otherwise. For HER-Dense, the agent receives a weighted sum of distance to the goal and contact and control costs as its reward. Humanoid Humanoid is a bipedal robot, with a 43-D state space and 17-D action space. The state contains information about the agent's height, orientation (yaw, pitch, roll), 3-D velocity, and the 3-D relative positions of each of its joints (knees, shoulders, etc.). The action consists of the torques at these joints. Minitaur Minitaur is a quadrupedal robot, with a 17-D state space and 8-D action space. The state contains information about motor angles, torques, velocities, and the orientation of the base. The action consists of the torque to be applied at each joint. Ant Ant is a quadrupedal robot, with a 111-D state space and 8-D action space. The state space contains the agent's height, 3-D linear and angular velocity, joint velocities, joint angles, and the external forces at each link. The action consists of the torques at all the 8 joints. Our choice of these locomotion environments is driven by the motivation provided earlier: we want an agent to navigate to a desired goal position in the 3-D space. In view of this, we do not use 2-D locomotion environments like Half-Cheetah, Walker, or Hopper. Also, though Reacher and Pusher are goal-based environments, we do not use them as they are manipulation environments, and we are aiming to achieve navigation by agents trained on 3-D locomotion tasks.
We show that the key to achieving good performance with IDMs lies in learning latent representations to encode the information shared between equivalent experiences, so that they can be generalized to unseen scenarios.
{"src": "scitldr-AIC_train", "paper_id": "HylloR4YDr"}