Imitation Learning: A Survey of Learning Methods <s> Optimization <s> This work reports experiments in interactive evolutionary robotics. The goal is to evolve an Artificial Neural Network (ANN) to control the locomotion of an 8-legged robot. The ANNs are encoded using a cellular developmental process called cellular encoding. In a previous work similar experiments have been carried on successfully on a simulated robot. They took however around 1,000,000 different ANN evaluations. In this work the fitness is determined on a real robot, and no more than a few hundreds evaluations can be performed. Various ideas were implemented so as to decrease the required number of evaluations from 1,000,000 to 200. First we used cell cloning and link typing. Second we did as many things as possible interactively: interactive problem decomposition, interactive syntactic constraints, interactive fitness. More precisely: 1- A modular design was chosen where a controller for an individual leg, with a precise neuronal interface was developed. 2- Syntactic constraints were used to promoting useful building blocs and impose an 8-fold symmetry. 3- We determine the fitness interactively by hand. We can reward features that would otherwise be very difficult to locate automatically. Interactive evolutionary robotics turns out to be quite successful, in the first bug-free run a global locomotion controller that is faster than a programmed controller could be evolved. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> The second order motion model is one of the fundamental questions, a mostly important object in motion planning research of mobile robots, especially in complex environment. Based on the research of the second order motion model, this paper puts forward a new method for adjusting robots to avoid obstacles in dynamic environment. A mathematical model is first established in which environmental information such as, destination of a mobile robot, velocity and direction of obstacles are considered. Secondly, a new particle swarm optimization (PSO) algorithm is used to search for solution of the multi-objective optimization problem as described in the mathematical model. Finally, by adjusting the velocity and direction of the mobile robot to avoid obstacles in real time, the robot can reach the goal safely. Simulation experiment shows that this method is better than tradition artificial potential field (APF) algorithm and its improved algorithm based on genetic algorithm for obstacle avoidance. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> The Robosoccer simulator is a challenging environment, where a human introduces a team of agents into a football virtual environment. Typically, agents are programmed by hand, but it would be a great advantage to transfer human experience into football agents. The first aim of this paper is to use machine learning techniques to obtain models of humans playing Robosoccer. These models can be used later to control a Robosoccer agent. However, models did not play as smoothly and optimally as the human. To solve this problem, the second goal of this paper is to incrementally correct models by means of evolutionary techniques, and to adapt them against more difficult opponents than the ones beatable by the human. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> Evolutionary algorithms are commonly used to create high-performing strategies or agents for computer games. In this paper, we instead choose to evolve the racing tracks in a car racing game. An evolvable track representation is devised, and a multiobjective evolutionary algorithm maximises the entertainment value of the track relative to a particular human player. This requires a way to create accurate models of players' driving styles, as well as a tentative definition of when a racing track is fun, both of which are provided. We believe this approach opens up interesting new research questions and is potentially applicable to commercial racing games. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented. <s> BIB005 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> The purpose of this paper is to search the best flight route efficiently for unmanned aerial vehicle (UAV) in the 3-dimention complicated topography. The proposed method for the best flight route is mainly utilizing evolutionary algorithm, and give the proper initial population of evolutionary algorithm through skeletonization, efficient pre-processing procedure. In order to provide a smooth flight route for UAV, this paper adopts B-spline Curve method. Several control points of B-spline Curve method must be determined to generate flight route. The best control points can be calculated by Particle Swarm Optimizer (PSO). In this paper, the initial population of PSO is provided by skeletonization. The skeletonization of pre-processing procedure mainly includes two parts: one is Skeletonization and the other is candidate path searching. The purpose of pre-processing procedure is to reduce computation time, to prevent search the best solutions aimless, and execute evolutionary process efficiently. This paper uses Matlab as the experiment environment. The results of the experiments present the proposed method can provide the best flight route for UAV efficiently. <s> BIB006 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> A new UAV path planning method based on ant colony optimization (ACO) is presented. The target position is considered as the food source which the ants are going to find. The enemy defense region is considered as the searching area of the ants and is divided into equally spaced grids. The ants move to the destination node through several nodes on the grid region. The visibility function of ACO algorithm considers the enemy threats intensity on the paths and the distance to the destination node. The weighted sums of the flight path length, the threat cost and the maximum restriction of the yaw angle are considered as the evaluation function of ACO algorithm. The pheromone amounts on the paths are updated according to the evaluation function values. Therefore, the UAV optimal flight path is expressed by a group of node number, which is obtained by the ants finding the optimal route to the food source. The ACO algorithm based UAV path planning method is characterized as simple coding and good optimization guidance, and the simulation results also show its effectiveness. <s> BIB007 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> Robot skill learning by imitation is an intuitive approach to learn robot skills from the observation of human behaviors. However, due to the discrepancy of mechanism between humans and robots such as the type of joint and the number of degree of freedom (DOF), a robot may not be able to imitate a human's movements faithfully. For a robot arm, the major problem is caused by the lack of a degree of freedom in the shoulder compared to a human arm. In this paper, we develop a similarity metric to evaluate how faithfully a robot arm imitates a human's arm movements. This metric is derived by utilizing the sequence-independent joint angle representation for both human and robot arms because it represents their postures more directly than the sequence-dependent Euler joint angle representation. In addition, the derived metric is formulated with the spatial relationship between human and robot arm postures instead of the Frobenius norm of the difference matrix between human and robot transformation matrices. To investigate the joint angles of the sequence-independent joint angle representation for a human arm, we adopt the particle-swarm optimization (PSO) to numerically derive them from human demonstration data. Computer simulations and experimental work were conducted to validate the proposed approach on a robot arm with two degrees of freedom in the shoulder and a DOF in the elbow. <s> BIB008 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> This paper reports an experience of use of a robotics educational kit to prototype a biped robot, the robot was used to validate a new walking trajectory generation approach. The use of robotic kits is an alternative for low-cost prototyping and testing robotic solutions. Even if these robots did not meet the requirements of an industrial design, they can offer a first validation alternative. In this work, the Bioloid robotic kit is used to implement and evaluate a new algorithm for biped gait generation called IK-PSO. The algorithm used a forward kinematics model and a target position of the biped center of mass, COM, to generate an inverse kinematics solution of the joints motions. The remaining of the control process is a classical one. The pre-computed position of the Centre of mass, COM, allowed the robot to walk with a high degree of confidence that its stability is guaranteed. The educational robotics kit used allowed a fast prototyping and a practical test of the proposed algorithm. <s> BIB009 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> Abstract We describe and compare several methods for generating game character controllers that mimic the playing style of a particular human player, or of a population of human players, across video game levels. Similarity in playing style is measured through an evaluation framework, that compares the play trace of one or several human players with the punctuated play trace of an AI player. The methods that are compared are either hand-coded, direct (based on supervised learning) or indirect (based on maximising a similarity measure). We find that a method based on neuroevolution performs best both in terms of the instrumental similarity measure and in phenomenological evaluation by human spectators. A version of the classic platform game “Super Mario Bros” is used as the testbed game in this study but the methods are applicable to other games that are based on character movement in space. <s> BIB010 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> Methodologies are emerging in many branches of computer science that demonstrate how human users and automated algorithms can collaborate on a problem such that their combined solutions outperform those produced by either humans or algorithms alone. The problem of behavior optimization in robotics seems particularly well-suited for this approach because humans have intuitions about how animals---and thus robots---should and should not behave, and can visually detect non-optimal behaviors that are trapped in local optima. Here we introduce a multiobjective approach in which a surrogate user (which stands in for a human user) deflects search away from local optima and a traditional fitness function eventually leads search toward the global optimum. We show that this approach produces superior solutions for a deceptive robotics problem compared to a similar search method that is guided by just a surrogate user or just a fitness function. <s> BIB011 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> Particle swarm optimization (PSO) is a heuristic global optimization method, proposed originally by Kennedy and Eberhart in 1995. It is now one of the most commonly used optimization techniques. This survey presented a comprehensive investigation of PSO. On one hand, we provided advances with PSO, including its modifications (including quantum-behaved PSO, bare-bones PSO, chaotic PSO, and fuzzy PSO), population topology (as fully connected, von Neumann, ring, star, random, etc.), hybridization (with genetic algorithm, simulated annealing, Tabu search, artificial immune system, ant colony algorithm, artificial bee colony, differential evolution, harmonic search, and biogeography-based optimization), extensions (to multiobjective, constrained, discrete, and binary optimization), theoretical analysis (parameter selection and tuning, and convergence analysis), and parallel implementation (in multicore, multiprocessor, GPU, and cloud computing forms). On the other hand, we offered a survey on applications of PSO to the following eight fields: electrical and electronic engineering, automation control systems, communication theory, operations research, mechanical engineering, fuel and energy, medicine, chemistry, and biology. It is hoped that this survey would be beneficial for the researchers studying PSO algorithms. <s> BIB012 </s> Imitation Learning: A Survey of Learning Methods <s> Optimization <s> Social learning plays an important role in behavior learning among social animals. In contrast to individual (asocial) learning, social learning has the advantage of allowing individuals to learn behaviors from others without incurring the costs of individual trials-and-errors. This paper introduces social learning mechanisms into particle swarm optimization (PSO) to develop a social learning PSO (SL-PSO). Unlike classical PSO variants where the particles are updated based on historical information, including the best solution found by the whole swarm (global best) and the best solution found by each particle (personal best), each particle in the proposed SL-PSO learns from any better particles (termed demonstrators) in the current swarm. In addition, to ease the burden of parameter settings, the proposed SL-PSO adopts a dimension-dependent parameter control method. The proposed SL-PSO is first compared with five representative PSO variants on 40 low-dimensional test functions, including shifted and rotated test functions. The scalability of the proposed SL-PSO is further tested by comparing it with five state-of-the-art algorithms for large-scale optimization on seven high-dimensional (100-D, 500-D, and 1000-D) benchmark functions. Our comparative results show that SL-PSO performs well on low-dimensional problems and is promising for solving large-scale problems as well. <s> BIB013
Optimization approaches can also be used to find a solution to a given problem. Definition 13. Given a cost function f : A → R that reflects the performance of an agent, where A is a set of input parameters and R is the set of real numbers, optimization methods aim to find the input parameters x 0 that minimize the cost function such that f (x 0 ) ≤ f (x) ∀x ∈ A. Similar to reinforcement learning, optimization techniques can be used to find solutions to problems by starting with a random solution and iteratively improving to optimize the fitness function. Evolutionary algorithms (EAs) are popular optimization methods that have extensively been used to find motor trajectories for robotic tasks  . EAs are used to generate motion trajectories for high and low DOF robots BIB009 BIB002 . Popular swarm intelligence methods such as Particle Swarm Optimization (PSO) BIB012 and Ant Colony Optimization (ACO) BIB007 ] are used to generate trajectories for unmanned vehicle navigation. These techniques simulate the behavior of living creatures to find an optimal global solution in the search space. As is the case with reinforcement learning, evolutionary algorithms can be integrated with imitation learning to improve trajectories learned by demonstration or to speed up the optimization process. In BIB005 , a genetic algorithm (GA) is used to optimize demonstrated motion trajectories. The trajectories are used as a starting population for the genetic algorithm. The recorded trajectories are encoded as chromosomes constituted of genes representing the motor primitives. The GA searches for the chromosome that optimizes a fitness function that evaluates the success of the task. Projecting the motor trajectories to a lower dimension illustrates the significant change between the optimized motion and the one learned directly from kinesthetic manipulation BIB005 . Similarly, in BIB003 , evolutionary algorithms are used after training agents in a soccer simulation. A possible solution (chromosome) is represented as a set of ifthen rules. The rules are finite due to the finite permutations of observations and actions. A weighted function of the number of goals and other performance measures is used to evaluate the fitness of a solution. Although the evolutionary algorithm had a small population size and did not employ crossover, it showed promising results over the rules learned from demonstrations. BIB004 also used evolutionary algorithms to optimize multiple objectives in a racing game. The algorithms evolve an optimized solution (controller) from an initial population of driving trajectories. Evaluation of the evolved controllers found that they stay faithful to the driving style of the players they are modeled after. This is true for quantitative measures such as speed and progress, and for subjective observations such as driving in the center of the road. BIB010 treat the weights of a neural network as the genome to be optimized. The initial population is provided by training the network with demonstrated samples to initialize the weights. The demonstrations are also used to create a fitness value corresponding to the mean squared error distance from the desired outputs (human actions). In BIB006 , Particle Swarm Optimization (PSO) is used to find the optimal path for an Unmanned Aerial Vehicle (UAV) by finding the best control points on a B-spline curve. The initial points that serve as the initial PSO particles are provided by skeletonization. A social variation of PSO is introduced in BIB013 , inspired by animals learning in nature from observing their peers. Each particle starts with a random solution and a fitness function is used to evaluate each solution. Then imitator particles (all except the one with the best fitness) modify their behavior by observing demonstrator particles (better-performing particles). As in nature, an imitator can learn from multiple demonstrators and a demonstrator can be used to teach more than one imitator. Interactive Evolutionary algorithms (IEAs) BIB001 ] employ a different paradigm. Rather than using human input to start an initial population of solutions and optimize them, IEA uses human input to judge the fitness of the solutions. To avoid the user evaluating too many potential solutions, a model is trained on supervised examples to estimate the human user's evaluation. In BIB011 , fitness-based search is combined with Preferencebased Policy Learning (PPL) to learn robot navigation. The user evaluations from PPL guide the search away from local minima, while the fitness-based search searches for a solution. In a similar spirit, BIB008 train a robot to imitate human arm movement. The difference in DOF between the human demonstrator and the robot obstructs using the demonstrations as an initial population. However, rather than using human input to subjectively evaluate a solution, the similarity of the robot movement to human demonstrations is quantitatively evaluated. A sequence-independent joint representation for the demonstrator and the learner is used to form a fitness function. PSO is used to find the joint angles to optimize this similarity measure. A different method of integrating demonstrations is proposed in El-Hussieny et al. [2015] . Inspired by Inverse Reinforcement Learning (see section on apprenticeship learning), an Inverse Linear Quadratic Regulator (ILQR) framework is used to learn cost function optimized by the human demonstrator. PSO is then employed to find a solution for the learned function instead of gradient methods.