Imitation Learning: A Survey of Learning Methods <s> Manually Designed Features <s> Summary This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It will be postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in form of movement primitives. It will be reviewed how research on representations of, and functional connections between action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning will also be described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical learning research. Parallels and differences between biological and computational approaches to imitation will be highlighted and an overview of current projects that actually employ imitation learning for humanoid robots will be given. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Manually Designed Features <s> Abstract This paper evaluates a model of human imitation of abstract, two-arm movements. The model consists of a hierarchy of artificial neural networks, which are abstractions of brain regions involved in visuo-motor control. The model is validated in a biomechanical simulation of a 37 degree of freedom (DOF) humanoid. Input to the model are data from human arm movements recorded using video and marker-based tracking systems. Results show a high qualitative and quantitative agreement with human data. The model’s reproduction is better or comparable to that of human subjects imitating the same movements. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Manually Designed Features <s> We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> Manually Designed Features <s> How does an individual use the knowledge ::: acquired through self exploration as a manipulable model through which to understand ::: others and benefit from their knowledge? ::: How can developmental and social learning be ::: combined for their mutual benefit? In this ::: paper we review a hierarchical architecture ::: (HAMMER) which allows a principled way ::: for combining knowledge through exploration ::: and knowledge from others, through the creation and use of multiple inverse and forward ::: models. We describe how Bayesian Belief Networks can be used to learn the association ::: between a robot’s motor commands and sensory consequences (forward models), and how ::: the inverse association can be used for imitation. Inverse models created through self ::: exploration, as well as those from observing ::: others can coexist and compete in a principled unified framework, that utilises the simulation theory of mind approach to mentally ::: rehearse and understand the actions of others. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> Manually Designed Features <s> Abstract We describe and compare several methods for generating game character controllers that mimic the playing style of a particular human player, or of a population of human players, across video game levels. Similarity in playing style is measured through an evaluation framework, that compares the play trace of one or several human players with the punctuated play trace of an AI player. The methods that are compared are either hand-coded, direct (based on supervised learning) or indirect (based on maximising a similarity measure). We find that a method based on neuroevolution performs best both in terms of the instrumental similarity measure and in phenomenological evaluation by human spectators. A version of the classic platform game “Super Mario Bros” is used as the testbed game in this study but the methods are applicable to other games that are based on character movement in space. <s> BIB005 </s> Imitation Learning: A Survey of Learning Methods <s> Manually Designed Features <s> A sequence of 3-D motion images of a human facing an imitator [i.e., a humanoid robot (HR)] is captured using a stereo-vision system installed on the HR. After acquiring enough motion sequences via image ratio and background registration, the estimation of seven feature points (i.e., the head, two arm tips, two leg tips, and two elbows) of the human are obtained. Using five feature points (i.e., the head, two arm tips, and two leg tips), the extraction of key posture frames from the recorded videos is achieved. To ensure a stable imitation motion of the HR, the human motions of the lower body (LB) and the upper body (UB) are imitated separately. All 11 stable motions for the LB are classified by the motion direction, the motion state, and the support phase, whereas only the two arm tips are employed to imitate the motions of the UB. Due to the disadvantages of traditional inverse kinematics (IK), neural-network-based IK is developed to reduce the computation time and improve the accuracy of the time response of the system. Finally, the combined motions of the LB and the UB with suitable interpolation and time intervals are applied to imitate the 3-D motion of a human. The corresponding experimental results confirm the efficacy of the proposed technique. <s> BIB006
These are features that are extracted using specially designed functions. These methods incorporate expert knowledge about the data and application to determine what useful information can be derived by processing the raw data. BIB003 extract features from the absolute positions of players on a soccer field. More meaningful information such as relative distance from a player, distance and angle to the goal, and length and angles of player triangles are calculated. The continuous features are discretized into overlapping tiles. This transformation of features from a numeric domain to binary tiles is reported to significantly improve learning. Manually designed features are popular with learning from visual data. They play an important part in computer vision methods that are used to teach machines by demonstration. Computer vision approaches are popular from an early point in teaching robots to behave from demonstrations BIB001 ]. These approaches rely on detecting objects of interest and tracking their movement to capture the demonstrated actions in a form that can be used for learning. In BIB004 , a computer vision system is employed to create representations that are used to train a Bayesian belief network. Training samples are generated by detecting and tracking objects in the scene, which is performed by clustering regions of the image according to their position and movement properties. BIB002 imitate human movement by tracking relevant points on the human body. A computer vision system is used to detect human arm movement and track motion based on markers worn by the performer. The system only learns when motion is detected by observing a change in the marker positions. In a recent study BIB006 , humanoid robots were trained to imitate human motion from visual observation. Demonstrations are captured using a stereo-vision system to create a 3D image sequence. The demonstrator's body is isolated from the sequence and a set of predetermined points on the upper and lower body are identified. Subsequently, the extracted features are used to estimate the demonstrator's posture along the trajectory. A variation of inverse kinematics that employs neural networks is used to reproduce the key posture features in the humanoid robot. For the Mario Bros game in , the source signal is the screenshot at the current frame. The image is divided into 22 × 22 equally sized cells. For each cell, 14 binary features (the value of each feature can be 0 or 1) are generated, each signifying whether or not the cell contains a certain object such as enemies, obstacles, and/or power-ups. As such, each cell can contain between 0 and 14 of the predefined objects. A demonstration is made up of the last four frames (so as to contain information about the direction in which objects are moving) as well as the last four chosen actions. BIB005 use a similar grid to represent the environment but add more numerical features and features describing the state of the character. Figure 3 illustrates dividing an image into subpatches and extracting binary features.