The optimization of functions to find the best solution according to one or several objectives has a central role in many engineering and research fields. Recently, a new family of optimization algorithms, named Quality-Diversity optimization, has been introduced, and contrasts with classic algorithms. Instead of searching for a single solution, Quality-Diversity algorithms are searching for a large collection of both diverse and high-performing solutions. The role of this collection is to cover the range of possible solution types as much as possible, and to contain the best solution for each type. The contribution of this paper is threefold. Firstly, we present a unifying framework of Quality-Diversity optimization algorithms that covers the two main algorithms of this family (Multi-dimensional Archive of Phenotypic Elites and the Novelty Search with Local Competition), and that highlights the large variety of variants that can be investigated within this family. Secondly, we propose algorithms with a new selection mechanism for Quality-Diversity algorithms that outperforms all the algorithms tested in this paper. Lastly, we present a new collection management that overcomes the erosion issues observed when using unstructured collections. These three contributions are supported by extensive experimental comparisons of Quality-Diversity algorithms on three different experimental scenarios.
Based on this observation, a variant of NSLC, named Behavioral Repertoire Evolution''(BR-Evolution @cite_36 ), has been introduced to progressively improve the archive's quality by replacing the solutions that are kept in the archive with better ones as soon as they are found. This approach has been applied to generate Behavioral Repertoires'' in robotics, which consists of a large collection of diverse, but effective, behaviors for a robotic agent in a single run of an evolutionary algorithm. It has also been used to produce collections of walking gaits, allowing a virtual six-legged robot to walk in every direction and at different speeds. The descriptor space is defined as the final position of the robot after walking for 3 seconds, while the quality score corresponds to an orientation error. As we reproduce this experiment in this paper, we provide additional descriptions and technical details in section .
@cite_36
1965675101
Learning in robotics typically involves choosing a simple goal (e.g. walking) and assessing the performance of each controller with regard to this task (e.g. walking speed). However, learning advanced, input-driven controllers (e.g. walking in each direction) requires testing each controller on a large sample of the possible input signals. This costly process makes difficult to learn useful low-level controllers in robotics. Here we introduce BR-Evolution, a new evolutionary learning technique that generates a behavioral repertoire by taking advantage of the candidate solutions that are usually discarded. Instead of evolving a single, general controller, BR-evolution thus evolves a collection of simple controllers, one for each variant of the target behavior; to distinguish similar controllers, it uses a performance objective that allows it to produce a collection of diverse but high-performing behaviors. We evaluated this new technique by evolving gait controllers for a simulated hexapod robot. Results show that a single run of the EA quickly finds a collection of controllers that allows the robot to reach each point of the reachable space. Overall, BR-Evolution opens a new kind of learning algorithm that simultaneously optimizes all the achievable behaviors of a robot. Numerous algorithms have been proposed to allow legged robots to learn to walk. However, most of these algorithms are devised to learn walking in a straight line, which is not sufficient to accomplish any real-world mission. Here we introduce the Transferability-based Behavioral Repertoire Evolution algorithm TBR-Evolution, a novel evolutionary algorithm that simultaneously discovers several hundreds of simple walking controllers, one for each possible direction. By taking advantage of solutions that are usually discarded by evolutionary processes, TBR-Evolution is substantially faster than independently evolving each controller. Our technique relies on two methods: 1 novelty search with local competition, which searches for both high-performing and diverse solutions, and 2 the transferability approach, which combines simulations and real tests to evolve controllers for a physical robot. We evaluate this new technique on a hexapod robot. Results show that with only a few dozen short experiments performed on the robot, the algorithm learns a repertoire of controllers that allows the robot to reach every point in its reachable space. Overall, TBR-Evolution introduced a new kind of learning algorithm that simultaneously optimizes all the achievable behaviors of a robot. The use of evolutionary robotics in robots with complex means of locomotion has, so far, mainly been limited to gait evolution. Increasing the number of degrees of freedom available to a controller significantly enlarges the search space, which in turn makes the evolution of solutions for a given task more challenging. In this paper, we present Evolutionary Repertoire-based Control (EvoRBC), an approach that enables the evolution of control for robots with arbitrary locomotion complexity. EvoRBC separates the synthesis of control into two levels: the generation of a repertoire of behavior primitives through the application of Quality Diversity techniques; and the evolution of a behavior arbitrator that uses the repertoire's primitives to solve a particular task. We evaluate EvoRBC in simulated robots with different numbers of degrees of freedom in two tasks, navigation and foraging. Our results show that while standard evolutionary approaches are highly affected by the locomotion complexity of the robot, EvoRBC is consistently able to evolve high-performing solutions. We also show that EvoRBC allows for the evolution of general controllers, that can be successfully used in robots different than those with which they were evolved. Most animals have significant behavioral expertise built in without having to explicitly learn it all from scratch. This expertise is a product of evolution of the organism; it can be viewed as a very long-term form of learning which provides a structured system within which individuals might learn more specialized skills or abilities. This paper suggests one possible mechanism for analagous robot evolution by describing a carefully designed series of networks, each one being a strict augmentation of the previous one, which control a six-legged walking machine capable of walking over rough terrain and following a person passively sensed in the infrared spectrum. As the completely decentralized networks are augmented, the robot's performance and behavior repertoire demonstrably improve. The rationale for such demonstrations is that they may provide a hint as to the requirements for automatically building massive networks to carry out complex sensory-motor tasks. The experiments with an actual robot ensure that an essence of reality is maintained and that no critical disabling problems have been ignored. Developing fast gaits for legged robots is a difficult task that requires optimizing parameters in a highly irregular, multidimensional space. In the past, walk optimization for quadruped robots, namely the Sony AIBO robot, was done by handtuning the parameterized gaits. In addition to requiring a lot of time and human expertise, this process produced sub-optimal results. Several recent projects have focused on using machine learning to automate the parameter search. Algorithms utilizing Powell's minimization method and policy gradient reinforcement learning have shown significant improvement over previous walk optimization results. In this paper we present a new algorithm for walk optimization based on an evolutionary approach. Unlike previous methods, our algorithm does not attempt to approximate the gradient of the multidimensional space. This makes it more robust to noise in parameter evaluations and avoids prematurely converging to local optima, a problem encountered by both of the previously suggested algorithms. Our evolutionary algorithm matches the best previous learning method, achieving several different walks of high quality. Furthermore, the best learned walks represent an impressive 20 improvement over our own best hand-tuned walks. While evolutionary computation and evolutionary robotics take inspiration from nature, they have long focused mainly on problems of performance optimization. Yet evolution in nature can be interpreted as more nuanced than a process of simple optimization. In particular, natural evolution is a divergent search that optimizes locally within each niche as it simultaneously diversifies. This tendency to discover both quality and diversity at the same time differs from many of the conventional algorithms of machine learning, and also thereby suggests a different foundation for inferring the approach of greatest potential for evolutionary algorithms. In fact, several recent evolutionary algorithms called quality diversity (QD) algorithms(e.g. novelty search with local competition and MAP-Elites) have drawn inspiration from this more nuanced view, aiming to fill a space of possibilities with the best possible example of each type of achievable behavior. The result is a new class of algorithms that return an archive of diverse, high-quality behaviors in a single run. The aim in this paper is to study the application of QD algorithms in challenging environments (in particular complex mazes) to establish their best practices for ambitious domains in the future. In addition to providing insight into cases when QD succeeds and fails, a new approach is investigated that hybridizes multiple views of behaviors (called behavior characterizations) in the same run, which succeeds in overcoming some of the challenges associated with searching for QD with respect to a behavior characterization that is not necessarily sufficient for generating both quality and diversity at the same time. This paper presents three feedback controllers that achieve an asymptotically stable, periodic, and fast walking gait for a 3-D bipedal robot consisting of a torso, revolute knees, and passive (unactuated) point feet. The walking surface is assumed to be rigid and flat; the contact between the robot and the walking surface is assumed to inhibit yaw rotation. The studied robot has 8 DOF in the single support phase and six actuators. In addition to the reduced number of actuators, the interest of studying robots with point feet is that the feedback control solution must explicitly account for the robot's natural dynamics in order to achieve balance while walking. We use an extension of the method of virtual constraints and hybrid zero dynamics (HZD), a very successful method for planar bipeds, in order to simultaneously compute a periodic orbit and an autonomous feedback controller that realizes the orbit, for a 3-D (spatial) bipedal walking robot. This method allows the computations for the controller design and the periodic orbit to be carried out on a 2-DOF subsystem of the 8-DOF robot model. The stability of the walking gait under closed-loop control is evaluated with the linearization of the restricted Poincare map of the HZD. Most periodic walking gaits for this robot are unstable when the controlled outputs are selected to be the actuated coordinates. Three strategies are explored to produce stable walking. The first strategy consists of imposing a stability condition during the search of a periodic gait by optimization. The second strategy uses an event-based controller to modify the eigenvalues of the (linearized) Poincare map. In the third approach, the effect of output selection on the zero dynamics is discussed and a pertinent choice of outputs is proposed, leading to stabilization without the use of a supplemental event-based controller. Learning algorithms are enabling robots to solve increasingly challenging real-world tasks. These approaches often rely on demonstrations and reproduce the behavior shown. Unexpected changes in the environment may require using different behaviors to achieve the same effect, for instance to reach and grasp an object in changing clutter. An emerging paradigm addressing this robustness issue is to learn a diverse set of successful behaviors for a given task, from which a robot can select the most suitable policy when faced with a new environment. In this paper, we explore a novel realization of this vision by learning a generative model over policies. Rather than learning a single policy, or a small fixed repertoire, our generative model for policies compactly encodes an unbounded number of policies and allows novel controller variants to be sampled. Leveraging our generative policy network, a robot can sample novel behaviors until it finds one that works for a new environment. We demonstrate this idea with an application of robust ball-throwing in the presence of obstacles. We show that this approach achieves a greater diversity of behaviors than an existing evolutionary approach, while maintaining good efficacy of sampled behaviors, allowing a Baxter robot to hit targets more often when ball throwing in the presence of obstacles. As part of the winning team in the Sony legged robot league of the RoboCup 2003 international robotic soccer competition, we used automatic gait optimisation. This was the first such technique seen in the competition and allowed an improvement in both speed and stability over previous gaits. Additionally, the technique was fast enough to run on-site at the competition, optimising the walk for the particular surface used. This paper describes the gait optimisation technique as well as some surprising features of the final gait discovered by the algorithm. Most animals exhibit significant neurological and morphological change throughout their lifetime. No robots to date, however, grow new morphological structure while behaving. This is due to technological limitations but also because it is unclear that morphological change provides a benefit to the acquisition of robust behavior in machines. Here I show that in evolving populations of simulated robots, if robots grow from anguilliform into legged robots during their lifetime in the early stages of evolution, and the anguilliform body plan is gradually lost during later stages of evolution, gaits are evolved for the final, legged form of the robot more rapidly—and the evolved gaits are more robust—compared to evolving populations of legged robots that do not transition through the anguilliform body plan. This suggests that morphological change, as well as the evolution of development, are two important processes that improve the automatic generation of robust behaviors for machines. It also provides an experimental platform for investigating the relationship between the evolution of development and robust behavior in biological organisms. The precise analysis of a patient's or an elderly person's walking pattern is very important for an effective intelligent active mobility assistance robot. This walking pattern can be described by a cyclic motion, which can be modeled using the consecutive gait phases. In this paper, we present a completely non-invasive framework for analyzing and recognizing a pathological human walking gait pattern. Our framework utilizes a laser range finder sensor to detect and track the human legs, and an appropriately synthesized Hidden Markov Model (HMM) for state estimation, and recognition of the gait patterns. We demonstrate the applicability of this setup using real data, collected from an ensemble of different elderly persons with a number of pathologies. The results presented in this paper demonstrate that the proposed human data analysis scheme has the potential to provide the necessary methodological (modeling, inference, and learning) framework for a cognitive behavior-based robot control system. More specifically, the proposed framework has the potential to be used for the classification of specific walking pathologies, which is needed for the development of a context-aware robot mobility assistant. This paper introduces a novel approach to representing and learning tool affordances by a robot. The tool representation described here uses a behavior-based approach to ground the tool affordances in the behavioral repertoire of the robot. The representation is learned during a behavioral babbling stage in which the robot randomly chooses different exploratory behaviors, applies them to the tool, and observes their effects on environmental objects. The paper shows how the autonomously learned affordance representation can be used to solve tool-using tasks by dynamically sequencing the exploratory behaviors based on their expected outcomes. The quality of the learned representation was tested on extension-of-reach tool-using tasks. This paper discusses a strategy for a humanoid robot to pull a fire hose while walking towards a desired position and orientation. A hybrid controller on the robot's wrist holding the fire hose is implemented for pulling it. The proposed controller can automatically determine the pulling force according to the robot's walking velocity. Through simulation analysis it is shown that when the robot walks while pulling the fire hose a drift in the walking direction is generated. To cope with this drift and to direct the robot to a desired position and orientation, a walking task is introduced. Using a motion capture system, the robot's chest position and orientation are monitored and feed to the robot's walking pattern generator to correct the orientation drift and to determine where to walk and when to stop walking. Through experimental results the validity of the proposed strategy was confirmed. It is shown that the proposed hybrid controller contributes to the improvement of the robot's balance when walking. In the area of biped robot research, much progress has been made in the past few years. However, some difficulties remain to be dealt with, particularly about the implementation of fast and dynamic walking gaits, in other words anthropomorphic gaits, especially on uneven terrain. In this perspective, both concepts of center of pressure (CoP) and zero moment point (ZMP) are obviously useful. In this paper, the two concepts are strictly defined, the CoP with respect to ground-feet contact forces, the ZMP with respect to gravity plus inertia forces. Then, the coincidence of CoP and ZMP is proven, and related control aspects are examined. Finally, a virtual CoP-ZMP is defined, allowing us to extend the concept when walking on uneven terrain. This paper is a theoretical study. Experimental results are presented in a companion paper, analyzing the evolution of the ground contact forces obtained from a human walker wearing robot feet as shoes. Robots have transformed many industries, most notably manufacturing, and have the power to deliver tremendous benefits to society, for example in search and rescue, disaster response, health care, and transportation. They are also invaluable tools for scientific exploration of distant planets or deep oceans. A major obstacle to their widespread adoption in more complex environments and outside of factories is their fragility. While animals can quickly adapt to injuries, current robots cannot “think outside the box” to find a compensatory behavior when they are damaged: they are limited to their pre-specified self-sensing abilities, which can diagnose only anticipated failure modes and strongly increase the overall complexity of the robot. In this thesis, we propose a different approach that considers having robots learn appropriate behaviors in response to damage. However, current learning techniques are slow even with small, constrained search spaces. To allow fast and creative adaptation, we combine the creativity of evolutionary algorithms with the learning speed of policy search algorithms through three contributions: the behavioral repertoires, the damage recovery using these repertoires and the transfer of knowledge across tasks. Globally, this work aims to provide the algorithmic foundations that will allow physical robots to be more robust, effective and autonomous. This paper presents a motion control design for a walking-assistant robot by combining passive-compliant behavior and active obstacle avoidance. The proposed method recognizes the intended motion for a user using gait movement and the applied force of a hand. A passive-compliant motion control is developed that allows the user to safely walk with the robot. The obstacle avoidance controller actively controls the velocity to guide the user in a direction that avoids collisions. A shared-control scheme combines active obstacle avoidance and passive-compliant motion commands. The experimental results verify the effectiveness of the proposed method and show that the robot allows a user to walk safely in a complex environment. A questionnaire survey of elderly users shows that the proposed design for a walking-assistant robot gives satisfactory performance. Legged locomotion on flowing ground (e.g., granular media) is unlike locomotion on hard ground because feet experience both solid- and fluid-like forces during surface penetration. Recent bioinspired legged robots display speed relative to body size on hard ground comparable with high-performing organisms like cockroaches but suffer significant performance loss on flowing materials like sand. In laboratory experiments, we study the performance (speed) of a small (2.3 kg) 6-legged robot, SandBot, as it runs on a bed of granular media (1-mm poppy seeds). For an alternating tripod gait on the granular bed, standard gait control parameters achieve speeds at best 2 orders of magnitude smaller than the 2 body lengths s (≈60 cm s) for motion on hard ground. However, empirical adjustment of these control parameters away from the hard ground settings restores good performance, yielding top speeds of 30 cm s. Robot speed depends sensitively on the packing fraction φ and the limb frequency ω, and a dramatic transition from rotary walking to slow swimming occurs when φ becomes small enough and or ω large enough. We propose a kinematic model of the rotary walking mode based on generic features of penetration and slip of a curved limb in granular media. The model captures the dependence of robot speed on limb frequency and the transition between walking and swimming modes but highlights the need for a deeper understanding of the physics of granular media. For an effective intelligent active mobility assistance robot, the walking pattern of a patient or an elderly person has to be analyzed precisely. A well-known fact is that the walking patterns are gaits, that is, cyclic patterns with several consecutive phases. These cyclic motions can be modeled using the consecutive gait phases. In this paper, we present a completely non-invasive framework for analyzing a normal human walking gait pattern. Our framework utilizes a laser range finder sensor to collect the data, a combination of filters to preprocess these data, and an appropriately synthesized Hidden Markov Model (HMM) for state estimation, and recognition of the gait data. We demonstrate the applicability of this setup using real data, collected from an ensemble of different persons. The results presented in this paper demonstrate that the proposed human data analysis scheme has the potential to provide the necessary methodological (modeling, inference, and learning) framework for a cognitive behavior-based robot control system. More specifically, the proposed framework has the potential to be used for the recognition of abnormal gait patterns and the subsequent classification of specific walking pathologies, which is needed for the development of a contextaware robot mobility assistant. To take full advantage of the flexibility of a modular robot system, users must be able to create and verify new configurations and behaviors quickly. We present a design framework that facilitates rapid creation of new configurations and behaviors through composition of existing ones, and tools to verify configurations and behaviors as they are being created. New configurations are created by combining existing sub-configurations, for example combining four legs and a body to create a walking robot. Behaviors are associated with each configuration, so that when sub-configurations are composed, their associated behaviors are immediately available for composition as well. We introduce a new motion description language (Series-Parallel Action Graphs) that facilitates the rapid creation of complex behaviors by composition of simpler behaviors. We provide tools that automatically verify configurations and behaviors during the design process, allowing the user to identify problems early and iterate quickly. In addition to verification, users can evaluate their configurations and behaviors in a physics-based simulator. Hybrid zero dynamics (HZD) has emerged as a popular framework for dynamic walking but has significant implementation difficulties when applied to the high degrees of freedom humanoids. The primary impediment is the process of gait design—it is difficult for optimizers to converge on a viable set of virtual constraints defining a gait. This paper presents a methodology that allows for fast and reliable generation of dynamic robotic walking gaits through the HZD framework, even in the presence of underactuation. Specifically, we describe an optimization formulation that builds upon the novel combination of HZD and direct collocation methods. Furthermore, achieving a scalable implementation required developing a defect-variable substitution formulation to simplify expressions, which ultimately allows us to generate compact analytic Jacobians of the constraints. We experimentally validate our methodology on an underactuated humanoid, DURUS, a spring-legged machine designed to facilitate energy-economical walking. We show that the optimization approach, in concert with the HZD framework, yields dynamic and stable walking gaits in hardware with a total electrical cost of transport of 1.33.