The optimization of functions to find the best solution according to one or several objectives has a central role in many engineering and research fields. Recently, a new family of optimization algorithms, named Quality-Diversity optimization, has been introduced, and contrasts with classic algorithms. Instead of searching for a single solution, Quality-Diversity algorithms are searching for a large collection of both diverse and high-performing solutions. The role of this collection is to cover the range of possible solution types as much as possible, and to contain the best solution for each type. The contribution of this paper is threefold. Firstly, we present a unifying framework of Quality-Diversity optimization algorithms that covers the two main algorithms of this family (Multi-dimensional Archive of Phenotypic Elites and the Novelty Search with Local Competition), and that highlights the large variety of variants that can be investigated within this family. Secondly, we propose algorithms with a new selection mechanism for Quality-Diversity algorithms that outperforms all the algorithms tested in this paper. Lastly, we present a new collection management that overcomes the erosion issues observed when using unstructured collections. These three contributions are supported by extensive experimental comparisons of Quality-Diversity algorithms on three different experimental scenarios.
Based on this observation, a variant of NSLC, named Behavioral Repertoire Evolution''(BR-Evolution @cite_36 ), has been introduced to progressively improve the archive's quality by replacing the solutions that are kept in the archive with better ones as soon as they are found. This approach has been applied to generate Behavioral Repertoires'' in robotics, which consists of a large collection of diverse, but effective, behaviors for a robotic agent in a single run of an evolutionary algorithm. It has also been used to produce collections of walking gaits, allowing a virtual six-legged robot to walk in every direction and at different speeds. The descriptor space is defined as the final position of the robot after walking for 3 seconds, while the quality score corresponds to an orientation error. As we reproduce this experiment in this paper, we provide additional descriptions and technical details in section .
@cite_36
2502984779 2034435325 2015630562 1965675101 2810627076 2624516165 2111065664 2738495647 2462548332 2035397042 2774807875 1977418904 2324567380 2281874712 2100097207 2523469089 2109008048 2033272336 2735089625 1910406352
The use of evolutionary robotics in robots with complex means of locomotion has, so far, mainly been limited to gait evolution. Increasing the number of degrees of freedom available to a controller significantly enlarges the search space, which in turn makes the evolution of solutions for a given task more challenging. In this paper, we present Evolutionary Repertoire-based Control (EvoRBC), an approach that enables the evolution of control for robots with arbitrary locomotion complexity. EvoRBC separates the synthesis of control into two levels: the generation of a repertoire of behavior primitives through the application of Quality Diversity techniques; and the evolution of a behavior arbitrator that uses the repertoire's primitives to solve a particular task. We evaluate EvoRBC in simulated robots with different numbers of degrees of freedom in two tasks, navigation and foraging. Our results show that while standard evolutionary approaches are highly affected by the locomotion complexity of the robot, EvoRBC is consistently able to evolve high-performing solutions. We also show that EvoRBC allows for the evolution of general controllers, that can be successfully used in robots different than those with which they were evolved. We conceived a novel notion of a signature to characterize evolvability.We conducted an extensive comparison of approaches to evolve robot gaits.A better evolvability signature corresponds to faster adaption time in new scenarios.Our evolvability measure is an analysis tool, beyond standard performance benchmarks. Evolutionary robotics is a promising approach to autonomously synthesize machines with abilities that resemble those of animals, but the field suffers from a lack of strong foundations. In particular, evolutionary systems are currently assessed solely by the fitness score their evolved artifacts can achieve for a specific task, whereas such fitness-based comparisons provide limited insights about how the same system would evaluate on different tasks, and its adaptive capabilities to respond to changes in fitness (e.g., from damages to the machine, or in new situations). To counter these limitations, we introduce the concept of "evolvability signatures", which picture the post-mutation statistical distribution of both behavior diversity (how different are the robot behaviors after a mutation?) and fitness values (how different is the fitness after a mutation?). We tested the relevance of this concept by evolving controllers for hexapod robot locomotion using five different genotype-to-phenotype mappings (direct encoding, generative encoding of open-loop and closed-loop central pattern generators, generative encoding of neural networks, and single-unit pattern generators (SUPG)). We observed a predictive relationship between the evolvability signature of each encoding and the number of generations required by hexapods to adapt from incurred damages. Our study also reveals that, across the five investigated encodings, the SUPG scheme achieved the best evolvability signature, and was always foremost in recovering an effective gait following robot damages. Overall, our evolvability signatures neatly complement existing task-performance benchmarks, and pave the way for stronger foundations for research in evolutionary robotics. Numerous algorithms have been proposed to allow legged robots to learn to walk. However, most of these algorithms are devised to learn walking in a straight line, which is not sufficient to accomplish any real-world mission. Here we introduce the Transferability-based Behavioral Repertoire Evolution algorithm TBR-Evolution, a novel evolutionary algorithm that simultaneously discovers several hundreds of simple walking controllers, one for each possible direction. By taking advantage of solutions that are usually discarded by evolutionary processes, TBR-Evolution is substantially faster than independently evolving each controller. Our technique relies on two methods: 1 novelty search with local competition, which searches for both high-performing and diverse solutions, and 2 the transferability approach, which combines simulations and real tests to evolve controllers for a physical robot. We evaluate this new technique on a hexapod robot. Results show that with only a few dozen short experiments performed on the robot, the algorithm learns a repertoire of controllers that allows the robot to reach every point in its reachable space. Overall, TBR-Evolution introduced a new kind of learning algorithm that simultaneously optimizes all the achievable behaviors of a robot. Learning in robotics typically involves choosing a simple goal (e.g. walking) and assessing the performance of each controller with regard to this task (e.g. walking speed). However, learning advanced, input-driven controllers (e.g. walking in each direction) requires testing each controller on a large sample of the possible input signals. This costly process makes difficult to learn useful low-level controllers in robotics. Here we introduce BR-Evolution, a new evolutionary learning technique that generates a behavioral repertoire by taking advantage of the candidate solutions that are usually discarded. Instead of evolving a single, general controller, BR-evolution thus evolves a collection of simple controllers, one for each variant of the target behavior; to distinguish similar controllers, it uses a performance objective that allows it to produce a collection of diverse but high-performing behaviors. We evaluated this new technique by evolving gait controllers for a simulated hexapod robot. Results show that a single run of the EA quickly finds a collection of controllers that allows the robot to reach each point of the reachable space. Overall, BR-Evolution opens a new kind of learning algorithm that simultaneously optimizes all the achievable behaviors of a robot. This research considers the task of evolving the physical structure of a robot to enhance its performance in various environments, which is a significant problem in the field of Evolutionary Robotics. Inspired by the fields of evolutionary art and sculpture, we evolve only targeted parts of a robot, which simplifies the optimisation problem compared to traditional approaches that must simultaneously evolve both (actuated) body and brain. Exploration fidelity is emphasised in areas of the robot most likely to benefit from shape optimisation, whilst exploiting existing robot structure and control. Our approach uses a Genetic Algorithm to optimise collections of Bezier splines that together define the shape of a legged robot's tibia, and leg performance is evaluated in parallel in a high-fidelity simulator. The leg is represented in the simulator as 3D-printable file, and as such can be readily instantiated in reality. Provisional experiments in three distinct environments show the evolution of environment-specific leg structures that are both high-performing and notably different to those evolved in the other environments. This proof-of-concept represents an important step towards the environment-dependent optimisation of performance-critical components for a range of ubiquitous, standard, and already-capable robots that can carry out a wide variety of tasks. Most animals have significant behavioral expertise built in without having to explicitly learn it all from scratch. This expertise is a product of evolution of the organism; it can be viewed as a very long-term form of learning which provides a structured system within which individuals might learn more specialized skills or abilities. This paper suggests one possible mechanism for analagous robot evolution by describing a carefully designed series of networks, each one being a strict augmentation of the previous one, which control a six-legged walking machine capable of walking over rough terrain and following a person passively sensed in the infrared spectrum. As the completely decentralized networks are augmented, the robot's performance and behavior repertoire demonstrably improve. The rationale for such demonstrations is that they may provide a hint as to the requirements for automatically building massive networks to carry out complex sensory-motor tasks. The experiments with an actual robot ensure that an essence of reality is maintained and that no critical disabling problems have been ignored. This paper proposes an evolutionary-group-based particle-swarm-optimization (EGPSO) algorithm for fuzzy-controller (FC) design. The EGPSO uses a group-based framework to incorporate crossover and mutation operations into particle-swarm optimization. The EGPSO dynamically forms different groups to select parents in crossover operations, particle updates, and replacements. An adaptive velocity-mutated operation (AVMO) is incorporated to improve search ability. The EGPSO is applied to design all of the free parameters in a zero-order Takagi-Sugeno-Kang (TSK)-type FC. The objective of EGPSO is to improve fuzzy-control accuracy and design efficiency. Comparisons with different population-based optimizations of fuzzy-control problems demonstrate the superiority of EGPSO performance. In particular, the EGPSO-designed FC is applied to mobile-robot navigation in unknown environments. In this application, the robot learns to follow object boundaries through an EGPSO-designed FC. A simple learning environment is created to build this behavior without an exhaustive collection of input-output training pairs in advance. A behavior supervisor is proposed to combine the boundary-following behavior and the target-seeking behavior for navigation, and the problem of dead cycles is considered. Successful mobile-robot navigation in simulation and real environments verifies the EGPSO-designed FC-navigation approach. Evolutionary algorithms have previously shown promise in generating controllers for legged robots. Multiple evaluations across many evolutionary generations are typically required — simulators are frequently used to accommodate this. However, performance degradation is frequently observed when transferring controllers from simulation to reality due to inconsistencies between the two. In this paper we demonstrate a testbed that permits repeated, direct evolution of hexapod controllers as a closed-loop system. The testbed uses a two-stage evolutionary process. In stage 1, a multi-objective evolutionary algorithm spreads a population of controllers across a space of desirable criteria. The second stage allows for specific criteria to be selected for on a per-mission basis, with promising initial controller parameters taken from the first stage. As the optimisation occurs directly on the robot, performance is guaranteed. Furthermore, controllers can be made specific to irregularities in e.g., motor wear, and robot mass distribution, creating controllers that are sensitive to the hardware state of the individual robot. While evolutionary computation and evolutionary robotics take inspiration from nature, they have long focused mainly on problems of performance optimization. Yet evolution in nature can be interpreted as more nuanced than a process of simple optimization. In particular, natural evolution is a divergent search that optimizes locally within each niche as it simultaneously diversifies. This tendency to discover both quality and diversity at the same time differs from many of the conventional algorithms of machine learning, and also thereby suggests a different foundation for inferring the approach of greatest potential for evolutionary algorithms. In fact, several recent evolutionary algorithms called quality diversity (QD) algorithms(e.g. novelty search with local competition and MAP-Elites) have drawn inspiration from this more nuanced view, aiming to fill a space of possibilities with the best possible example of each type of achievable behavior. The result is a new class of algorithms that return an archive of diverse, high-quality behaviors in a single run. The aim in this paper is to study the application of QD algorithms in challenging environments (in particular complex mazes) to establish their best practices for ambitious domains in the future. In addition to providing insight into cases when QD succeeds and fails, a new approach is investigated that hybridizes multiple views of behaviors (called behavior characterizations) in the same run, which succeeds in overcoming some of the challenges associated with searching for QD with respect to a behavior characterization that is not necessarily sufficient for generating both quality and diversity at the same time. Genetic Network Programming (GNP) has been proposed as one of the evolutionary algorithms and extended with reinforcement learning (GNP-RL). The combination of evolution and learning can efficiently evolve programs and the fitness improvement has been confirmed in the simulations of tileworld problems, elevator group supervisory control systems, stock trading models and wall following behavior of Khepera robot. However, its robustness in testing environments has not been analyzed in detail yet. In this paper, the learning mechanism in the testing environment is introduced and it is confirmed that GNP-RL can show the robustness using a robot simulator WEBOTS, especially when unexperienced sensor troubles suddenly occur. The simulation results show that GNP-RL works well in the testing even if wrong sensor information is given because GNP-RL has a function to change programs using alternative actions automatically. In addition, the analysis on the effects of the parameters of GNP-RL is carried out in both training and testing simulations. Consider a setting where robots must visit sites represented as nodes in a graph, but each robot may fail when traversing an edge. The goal is to find a set of paths for a team of robots which maximizes the expected number of nodes collectively visited, while guaranteeing that the paths satisfy a notion of “independence” formalized by a matroid (e.g. limits on team size, number of visits to regions), and that the probabilities that each robot survives to its destination are above a given threshold. We call this problem the Matroid Team Surviving Orienteers (MTSO) problem, which has broad applications such as environmental monitoring in risky regions and search and rescue in dangerous conditions. We present the MTSO formally and detail numerous examples of matroids in a path planning context. We then propose an approximate greedy algorithm for selecting a feasible set of paths and prove that the value of the output is within a factor p s p s + λ of the optimum, where p s is the per-robot survival probability threshold and 1 λ ≤ 1 is the approximation factor of an oracle routine for the well known orienteering problem. We demonstrate the efficiency of our approach by applying it to a scenario where a team of robots must gather information while avoiding pirates in the Coral Triangle. Legged robots can potentially venture beyond the limits of wheeled vehicles. While creating controllers for such robots by hand is possible, evolutionary algorithms are an alternative that can reduce the burden of hand-crafting robotic controllers. Although major evolutionary approaches to legged locomotion can generate oscillations through popular techniques such as continuous time recurrent neural networks (CTRNNs) or sinusoidal input, they typically face a challenge in maintaining long-term stability. The aim of this paper is to address this challenge by introducing an effective alternative based on a new type of neuron called a single-unit pattern generator (SUPG). The SUPG, which is indirectly encoded by a compositional pattern producing network (CPPN) evolved by HyperNEAT, produces a flexible temporal activation pattern that can be reset and repeated at any time through an explicit trigger input, thereby allowing it to dynamically recalibrate over time to maintain stability. The SUPG approach, which is compared to CTRNNs and sinusoidal input, is shown to produce natural-looking gaits that exhibit superior stability over time, thereby providing a new alternative for evolving oscillatory locomotion. Received Feb 6, 2012 Revised Mar 12, 2012 Accepted Mar 21, 2012 This paper proposes a behavior-switching control strategy of an evolutionary robotics based on Artificial Neural Network (ANN) and Genetic Algorithms (GA). This method is able not only to construct the reinforcement learning models for autonomous robots and evolutionary robot modules that control behaviors and reinforcement learning environments, and but also to perform the behavior-switching control and obstacle avoidance of an evolutionary robotics (ER) in time-varying environments with static and moving obstacles by combining ANN and GA. The experimental results on the basic behaviors and behavior-switching control have demonstrated that our method can perform the decision-making strategy and parameters set optimization of FNN and GA by learning and can escape successfully from the trap of a local minima and avoid “motion deadlock” status of humanoid soccer robotics agents, and reduce the oscillation of the planned trajectory between the multiple obstacles by crossover and mutation. Some results of the proposed algorithm have been successfully applied to our simulation humanoid robotics soccer team CIT3D which won the 1st prize of RoboCup Championship and China Open 2010 (July 2010) and the 2nd place of the official Robo Cup World Championship (2011) on 5-11 July, 2011 in Istanbul, Turkey. As compared with the conventional behavior network and the adaptive behavior method, the genetic encoding complexity of our algorithm is simplified, and the network performance and the convergence rate ρ have been greatly improved. Keyword: For robotic systems to realise lifelong autonomy they must be able to navigate accurately in changing environments. In this thesis we describe, implement and validate a new approach to the problem of long-term navigation. To begin, we present our stereo visual odometry system which provides highly accurate pose estimation. Our approach combines several techniques found in existing implementations and a recently published image descriptor that simplifies the solution architecture. The performance and versatility of our system is demonstrated through testing on multiple datasets. Equipped with our visual odometry system, we describe a new approach to the problem of lifelong navigation. We learn a model whose complexity varies naturally in accordance with the variation of scene appearance. As the robot repeatedly traverses its workspace, it accumulates distinct visual experiences that, in concert, implicitly represent the scene variation - each experience captures a visual mode. When operating in a previously visited area, we continually try to localise in these previous experiences while simultaneously running the visual odometry. Failure to localise in a sufficient number of prior experiences indicates an insufficient model of the workspace and instigates the laying down of the live image sequence as a new distinct experience. In this way, over time we can capture the typical temporally varying appearance of an environment and the number of experiences required tends to a constant. Although we focus on vision as a primary sensor, the ideas we present here are equally applicable to other sensor modalities. We demonstrate our approach working on a road vehicle operating over a three month period at different times of day, in different weather and lighting conditions. Several researchers have demonstrated how complex action sequences can be learned through neuroevolution (i.e., evolving neural networks with genetic algorithms). However, complex general behavior such as evading predators or avoiding obstacles, which is not tied to specific environments, turns out to be very difficult to evolve. Often the system discovers mechanical strategies, such as moving back and forth, that help the agent cope but are not very effective, do not appear believable, and do not generalize to new environments. The problem is that a general strategy is too difficult for the evolution system to discover directly. This article proposes an approach wherein such complex general behavior is learned incrementally, by starting with simpler behavior and gradually making the task more challenging and general. The task transitions are implemented through successive stages of Delta coding (i.e., evolving modifications), which allows even converged populations to adapt to the new task. The method is... As a new way of training generative models, Generative Adversarial Nets (GAN) that uses a discriminative model to guide the training of the generative model has enjoyed considerable success in generating real-valued data. However, it has limitations when the goal is for generating sequences of discrete tokens. A major reason lies in that the discrete outputs from the generative model make it difficult to pass the gradient update from the discriminative model to the generative model. Also, the discriminative model can only assess a complete sequence, while for a partially generated sequence, it is non-trivial to balance its current score and the future one once the entire sequence has been generated. In this paper, we propose a sequence generation framework, called SeqGAN, to solve the problems. Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update. The RL reward signal comes from the GAN discriminator judged on a complete sequence, and is passed back to the intermediate state-action steps using Monte Carlo search. Extensive experiments on synthetic data and real-world tasks demonstrate significant improvements over strong baselines. We present a learning system which is able to quickly and reliably acquire a robust feedback control policy for 3D dynamic walking from a blank-slate using only trials implemented on our physical robot. The robot begins walking within a minute and learning converges in approximately 20 minutes. This success can be attributed to the mechanics of our robot, which are modeled after a passive dynamic walker, and to a dramatic reduction in the dimensionality of the learning problem. We reduce the dimensionality by designing a robot with only 6 internal degrees of freedom and 4 actuators, by decomposing the control system in the frontal and sagittal planes, and by formulating the learning problem on the discrete return map dynamics. We apply a stochastic policy gradient algorithm to this reduced problem and decrease the variance of the update using a state-based estimate of the expected cost. This optimized learning system works quickly enough that the robot is able to continually adapt to the terrain as it walks. In the context of developmental robotics, a robot has to cope with complex sensorimotor spaces by reducing their dimensionality. In the case of sensor space reduction, classical approaches for pattern recognition use either hard- coded feature detection or supervised learning. We believe supervised learning and hard-coded feature extraction must be extended with unsupervised learning of feature representations. In this paper, we present an approach to learn represen- tations using space-variant images and saccades. The saccades are driven by a measure of quantity of information in the visual scene, emerging from the activations of Restricted Boltzmann Machines (RBMs). The RBM, a generative model, is trained incrementally on locations where the system saccades. Our approach is implemented using real data captured by a NAO robot in indoor conditions. I. INTRODUCTION Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment. Evolutionary Robotics allows robots with limited sensors and processing to tackle complex tasks by means of sensory-motor coordination. In this article we show the first application of the Behavior Tree framework on a real robotic platform using the evolutionary robotics methodology. This framework is used to improve the intelligibility of the emergent robotic behavior over that of the traditional neural network formulation. As a result, the behavior is easier to comprehend and manually adapt when crossing the reality gap from simulation to reality. This functionality is shown by performing real-world flight tests with the 20-g DelFly Explorer flapping wing micro air vehicle equipped with a 4-g onboard stereo vision system. The experiments show that the DelFly can fully autonomously search for and fly through a window with only its onboard sensors and processing. The success rate of the optimized behavior in simulation is 88 , and the corresponding real-world performance is 54 after user adaptation. Although this leaves room for improvement, it is higher than the 46 success rate from a tuned user-defined controller.