Programming a robot to deal with open-ended tasks remains a challenge, in particular if the robot has to manipulate objects. Launching, grasping, pushing or any other object interaction can be simulated but the corresponding models are not reversible and the robot behavior thus cannot be directly deduced. These behaviors are hard to learn without a demonstration as the search space is large and the reward sparse. We propose a method to autonomously generate a diverse repertoire of simple object interaction behaviors in simulation. Our goal is to bootstrap a robot learning and development process with limited informations about what the robot has to achieve and how. This repertoire can be exploited to solve different tasks in reality thanks to a proposed adaptation method or could be used as a training set for data-hungry algorithms. The proposed approach relies on the definition of a goal space and generates a repertoire of trajectories to reach attainable goals, thus allowing the robot to control this goal space. The repertoire is built with an off-the-shelf simulation thanks to a quality diversity algorithm. The result is a set of solutions tested in simulation only. It may result in two different problems: (1) as the repertoire is discrete and finite, it may not contain the trajectory to deal with a given situation or (2) some trajectories may lead to a behavior in reality that differs from simulation because of a reality gap. We propose an approach to deal with both issues by using a local linearization between the motion parameters and the observed effects. Furthermore, we present an approach to update the existing solution repertoire with the tests done on the real robot. The approach has been validated on two different experiments on the Baxter robot: a ball launching and a joystick manipulation tasks.
Several methods have been put forward to make robot autonomously explore their environment and learn new skills to reach various goals. An important part of this literature treats this as a control problem, where the system learns an inverse model (or multiple inverse models) to generate the motor commands in order to reach an arbitrary position in a given goal space. Following some works showing that human babies learn how to control their body through body babbling'' @cite_9 , several methods use a similar motor babbling approach in robots with random motor orders @cite_16 @cite_23 . The two main issues of these approaches are that (1) learning the inverse model from collected samples is a challenging supervised machine learning problem for non-toy problems and that (2) sampling the motor space - which may be large - in a sample-efficient way is required and also difficult to achieve. Several approaches have been proposed to tackle these issues.
@cite_9 @cite_16 @cite_23
2587621706 2735089625 2004303440 2052519881
Developmental robotics suggests that the forward and inverse kinematics should be learned through a sensory-motor mapping, instead of being programmed in advance. Motor babbling and goal babbling are two common approaches to generate training samples used to acquire a sensory-motor mapping. Motor babbling typically needs a considerable amount of training data and time to acquire a sufficient mapping, while goal babbling poses difficulties on how to select appropriate goals. In this paper, we propose a neurobiologically-inspired system to progressively learn a sensory-motor mapping bootstrapped from a simple constrained DOF exploration, which generates much less training data than motor babbling. Our proposed system is designed according to two neurobiologically-inspired paradigms: spatiotemporal prediction and uniformity. The spatiotemporal prediction capability facilitates the acquisition of sensory-motor mappings with less amount of training data on the one hand, and facilitates robust behaviour on the other hand. The uniform system design structure is the foundation for building a scalable architecture for cognitive development. We use an improved version of our predictive action selector (PAS) as building block of our system. We validate a PAS on a 2 DOF robot head where the robot learns object tracking and evading. Then we validate a second PAS on a 5 DOF arm where it learns reaching. Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment. We introduce the Self-Adaptive Goal Generation Robust Intelligent Adaptive Curiosity (SAGG-RIAC) architecture as an intrinsically motivated goal exploration mechanism which allows active learning of inverse models in high-dimensional redundant robots. This allows a robot to efficiently and actively learn distributions of parameterized motor skills policies that solve a corresponding distribution of parameterized tasks goals. The architecture makes the robot sample actively novel parameterized tasks in the task space, based on a measure of competence progress, each of which triggers low-level goal-directed learning of the motor policy parameters that allow to solve it. For both learning and generalization, the system leverages regression techniques which allow to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously learnt correspondences between policy and task parameters. We present experiments with high-dimensional continuous sensorimotor spaces in three different robotic setups: (1) learning the inverse kinematics in a highly-redundant robotic arm, (2) learning omnidirectional locomotion with motor primitives in a quadruped robot, and (3) an arm learning to control a fishing rod with a flexible wire. We show that (1) exploration in the task space can be a lot faster than exploration in the actuator space for learning inverse models in redundant robots; (2) selecting goals maximizing competence progress creates developmental trajectories driving the robot to progressively focus on tasks of increasing complexity and is statistically significantly more efficient than selecting tasks randomly, as well as more efficient than different standard active motor babbling methods; (3) this architecture allows the robot to actively discover which parts of its task space it can learn to reach and which part it cannot. We present a neural network approach to early motor learning. The goal is to explore the needs for boot-strapping the control of hand movements in a biologically plausible learning scenario. The model is applied to the control of hand postures of the humanoid robot ASIMO by means of full upper body movements. For training, we use an efficient online scheme for recurrent reservoir networks consisting of supervised backpropagation-decorrelation output adaptation and an unsupervised intrinsic plasticity reservoir optimization. We demonstrate that the network can acquire accurate inverse models for the highly redundant ASIMO, applying bi-manual target motions and exploiting all upper body degrees of freedom. We show that very few, but highly symmetric training data is sufficient to generate excellent generalization capabilities to untrained target motions. We also succeed in reproducing real motion recorded from a human demonstrator, massively differing from the training data in range and dynamics. The demonstrated generalization capabilities provide a fundamental prerequisite for an autonomous and incremental motor learning in an developmentally plausible way. Our exploration process - though not yet fully autonomous - clearly shows that goal-directed exploration can, in contrast to “babbling” of joints angles, be done very efficiently even for many degrees of freedom and non-linear kinematic configurations as ASIMOs.