As demand drives systems to generalize to various domains and problems, the study of multitask, transfer and lifelong learning has become an increasingly important pursuit. In discrete domains, performance on the Atari game suite has emerged as the de facto benchmark for assessing multitask learning. However, in continuous domains there is a lack of agreement on standard multitask evaluation environments which makes it difficult to compare different approaches fairly. In this work, we describe a benchmark set of tasks that we have developed in an extendable framework based on OpenAI Gym. We run a simple baseline using Trust Region Policy Optimization and release the framework publicly to be expanded and used for the systematic comparison of multitask, transfer, and lifelong learning in continuous domains.
Several works investigate multitask or transfer learning with MuJoCo tasks. These tasks include: navigating around a wall (where a wall separates an agent from its goal); the OpenAI Gym Reacher environment with an added image state space of the environment; jumping over a wall using a model similar to the OpenAI Half-Cheetah environment @cite_4 ; varying the gravity of various standard OpenAI Gym benchmark environments (Reacher, Hopper, Humanoid, HalfCheetah) and transferring between the modified environments; adding motor noise to the same set of environments @cite_8 ; simulated grasping and stacking using a Jaco arm @cite_6 ; and several custom grasping and manipulation tasks to demonstrate learning invariant feature spaces @cite_13 .
@cite_13 @cite_4 @cite_6 @cite_8
2786036274 2883192382 2952996256 2028249667
In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time, which is already a problem in single task learning. We have developed a new distributed agent IMPALA (Importance-Weighted Actor Learner Architecture) that can scale to thousands of machines and achieve a throughput rate of 250,000 frames per second. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace, which was critical for achieving learning stability. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (, 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (, 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents, use less data and crucially exhibits positive transfer between tasks as a result of its multi-task approach. We present Toribash Learning Environment (ToriLLE), a learning environment for machine learning agents based on the video game Toribash. Toribash is a MuJoCo-like environment of two humanoid character fighting each other hand-to-hand, controlled by changing actuation modes of the joints. Competitive nature of Toribash as well its focused domain provide a platform for evaluating self-play methods, and evaluating machine learning agents against human players. In this paper we describe the environment with ToriLLE's capabilities and limitations, and experimentally show its applicability as a learning environment. The source code of the environment and conducted experiments can be found at this https URL. One approach to Imitation Learning is Behavior Cloning, in which a robot observes a supervisor and infers a control policy. A known problem with this "off-policy" approach is that the robot's errors compound when drifting away from the supervisor's demonstrations. On-policy, techniques alleviate this by iteratively collecting corrective actions for the current robot policy. However, these techniques can be tedious for human supervisors, add significant computation burden, and may visit dangerous states during training. We propose an off-policy approach that injects noise into the supervisor's policy while demonstrating. This forces the supervisor to demonstrate how to recover from errors. We propose a new algorithm, DART (Disturbances for Augmenting Robot Trajectories), that collects demonstrations with injected noise, and optimizes the noise level to approximate the error of the robot's trained policy during data collection. We compare DART with DAgger and Behavior Cloning in two domains: in simulation with an algorithmic supervisor on the MuJoCo tasks (Walker, Humanoid, Hopper, Half-Cheetah) and in physical experiments with human supervisors training a Toyota HSR robot to perform grasping in clutter. For high dimensional tasks like Humanoid, DART can be up to @math faster in computation time and only decreases the supervisor's cumulative reward by @math during training, whereas DAgger executes policies that have @math less cumulative reward than the supervisor. On the grasping in clutter task, DART obtains on average a @math performance increase over Behavior Cloning. This paper proposes evolutionary wall-following control of a mobile robot using an interval type-2 fuzzy controller (IT2FC) with species-differential-evolution-activated continuous ant colony optimization (SDE-CACO). Both the position and speed of a mobile robot are controlled by using two IT2FCs to improve noise resistance ability. A new cost function is defined to accurately evaluate the wall-following performance of an evolutionary IT2FC. A two-stage training approach is proposed that learns a position IT2FC followed by a speed IT2FC to optimize both the wall-following accuracy and the moving speed. The proposed learning approach avoids the time consuming task of the exhaustive collection of supervised input-output training pairs. All fuzzy rules are generated online using a clustering-based approach during the evolutionary learning process. All of the free parameters in an online-generated IT2FC are optimized using SDE-CACO, in which an SDE mutation operation is incorporated within a continuous ACO to improve its explorative ability. The proposed SDE-CACO is compared with various population-based optimization algorithms to demonstrate its efficiency and effectiveness in the wall-following control problem. This study also includes experiments that demonstrate wall-following control utilizing a real mobile robot.