Programming a robot to deal with open-ended tasks remains a challenge, in particular if the robot has to manipulate objects. Launching, grasping, pushing or any other object interaction can be simulated but the corresponding models are not reversible and the robot behavior thus cannot be directly deduced. These behaviors are hard to learn without a demonstration as the search space is large and the reward sparse. We propose a method to autonomously generate a diverse repertoire of simple object interaction behaviors in simulation. Our goal is to bootstrap a robot learning and development process with limited informations about what the robot has to achieve and how. This repertoire can be exploited to solve different tasks in reality thanks to a proposed adaptation method or could be used as a training set for data-hungry algorithms. The proposed approach relies on the definition of a goal space and generates a repertoire of trajectories to reach attainable goals, thus allowing the robot to control this goal space. The repertoire is built with an off-the-shelf simulation thanks to a quality diversity algorithm. The result is a set of solutions tested in simulation only. It may result in two different problems: (1) as the repertoire is discrete and finite, it may not contain the trajectory to deal with a given situation or (2) some trajectories may lead to a behavior in reality that differs from simulation because of a reality gap. We propose an approach to deal with both issues by using a local linearization between the motion parameters and the observed effects. Furthermore, we present an approach to update the existing solution repertoire with the tests done on the real robot. The approach has been validated on two different experiments on the Baxter robot: a ball launching and a joystick manipulation tasks.
A proposal to make the sampling process more efficient is to sample in the goal space rather than in the motor space, choosing points in the goal space and using the robot's current inverse model to to reach them, thus generating new samples that allow to further train the inverse model online. This goal babbling'' approach has been found to be more sample-efficient than motor babbling when using a small set of simple predefined goal space targets @cite_40 , or random goals that can be generated without any prior knowledge of the expected robot behavior @cite_21 . Since an existing inverse model is required to generate motions at any time, this model has first to be bootstrapped, for example with random initialization. Further developments of this goal babbling approach improve the sampling efficiency through the use of intrinsic motivations that choose goals maximizing the learning progress @cite_25 .
@cite_40 @cite_21 @cite_25
2587621706 2233446961 2004303440 2810132790
Developmental robotics suggests that the forward and inverse kinematics should be learned through a sensory-motor mapping, instead of being programmed in advance. Motor babbling and goal babbling are two common approaches to generate training samples used to acquire a sensory-motor mapping. Motor babbling typically needs a considerable amount of training data and time to acquire a sufficient mapping, while goal babbling poses difficulties on how to select appropriate goals. In this paper, we propose a neurobiologically-inspired system to progressively learn a sensory-motor mapping bootstrapped from a simple constrained DOF exploration, which generates much less training data than motor babbling. Our proposed system is designed according to two neurobiologically-inspired paradigms: spatiotemporal prediction and uniformity. The spatiotemporal prediction capability facilitates the acquisition of sensory-motor mappings with less amount of training data on the one hand, and facilitates robust behaviour on the other hand. The uniform system design structure is the foundation for building a scalable architecture for cognitive development. We use an improved version of our predictive action selector (PAS) as building block of our system. We validate a PAS on a 2 DOF robot head where the robot learns object tracking and evading. Then we validate a second PAS on a 5 DOF arm where it learns reaching. Enabling robots to accomplish sophisticated tasks requires enhancing their capability to plan at multiple levels of discrete and continuous abstractions. Toward this goal, the proposed approach couples the ability of sampling-based motion planning to handle the complexity arising from high-dimensional robotic systems, nonlinear dynamics, and collision avoidance with the ability of discrete planning to handle discrete specifications. The approach makes it possible to specify tasks via Linear Temporal Logic (LTL) and automatically computes collision-free and dynamically-feasible motions that enable the robot to carry out assigned tasks. While discrete planning guides sampling-based motion planning, the latter feeds back information to further refine the guide and advance the search. Sampling is also used in the discrete space to shorten the length of the discrete plans and to expand the search toward new discrete states. Experiments with high-dimensional dynamical robot models performing various LTL tasks show significant computational speedups over related work. We introduce the Self-Adaptive Goal Generation Robust Intelligent Adaptive Curiosity (SAGG-RIAC) architecture as an intrinsically motivated goal exploration mechanism which allows active learning of inverse models in high-dimensional redundant robots. This allows a robot to efficiently and actively learn distributions of parameterized motor skills policies that solve a corresponding distribution of parameterized tasks goals. The architecture makes the robot sample actively novel parameterized tasks in the task space, based on a measure of competence progress, each of which triggers low-level goal-directed learning of the motor policy parameters that allow to solve it. For both learning and generalization, the system leverages regression techniques which allow to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously learnt correspondences between policy and task parameters. We present experiments with high-dimensional continuous sensorimotor spaces in three different robotic setups: (1) learning the inverse kinematics in a highly-redundant robotic arm, (2) learning omnidirectional locomotion with motor primitives in a quadruped robot, and (3) an arm learning to control a fishing rod with a flexible wire. We show that (1) exploration in the task space can be a lot faster than exploration in the actuator space for learning inverse models in redundant robots; (2) selecting goals maximizing competence progress creates developmental trajectories driving the robot to progressively focus on tasks of increasing complexity and is statistically significantly more efficient than selecting tasks randomly, as well as more efficient than different standard active motor babbling methods; (3) this architecture allows the robot to actively discover which parts of its task space it can learn to reach and which part it cannot. Intrinsically motivated goal exploration processes enable agents to autonomously sample goals to explore efficiently complex environments with high-dimensional continuous actions. They have been applied successfully to real world robots to discover repertoires of policies producing a wide diversity of effects. Often these algorithms relied on engineered goal spaces but it was recently shown that one can use deep representation learning algorithms to learn an adequate goal space in simple environments. However, in the case of more complex environments containing multiple objects or distractors, an efficient exploration requires that the structure of the goal space reflects the one of the environment. In this paper we show that using a disentangled goal space leads to better exploration performances than an entangled goal space. We further show that when the representation is disentangled, one can leverage it by sampling goals that maximize learning progress in a modular manner. Finally, we show that the measure of learning progress, used to drive curiosity-driven exploration, can be used simultaneously to discover abstract independently controllable features of the environment.