Imitation Learning: A Survey of Learning Methods <s> Hierarchical Models <s> Abstract This paper presents a framework that gives robots the ability to initially learn a task behavior from observing others. The framework includes a method for the robots to increase performance while operating in the task environment. We demonstrate this approach applied to air hockey and the marble maze task. Our robots initially learn to perform the tasks using learning from observation, and then increase their performance through practice. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Hierarchical Models <s> Humanoid robots working alongside humans in everyday environments is a long standing goal of the robotics community. To achieve this goal, methods for developing new robot behaviors that are intuitive and accessible to non-programmers are required. In this paper, we present a demonstration-based method for teaching distributed autonomous robots to coordinate their actions and perform collaborative multi-robot tasks. Within the presented framework, each robot learns an individual policy from teacher demonstrations using a confidence-based algorithm. Based on this learning approach, we contribute three techniques for teaching multi-robot coordination using different information sharing strategies. We evaluate and compare these approaches by teaching two Sony QRIO humanoid robots to perform three collaborative ball sorting tasks. <s> BIB002
Rather than using a single model to reproduce human behavior, a hierarchical model can be employed that breaks down the learned actions into a number of phases. A classification model can be used to decide which action or subaction, from a set of possible actions, should be performed at a given time. A different model is then used to define the details of the selected action, where each possible low-level action has a designated model. Figure 4 Illustrates an example of hierarchical actions. BIB001 use a hierarchical approach for imitation learning on two different problems. The first is air hockey, which is played against an opponent, and the objective is to shoot a puck into the opponent's goal while protecting your own. The second game is marble maze; the maze can be tilted around different axes to move the ball toward the end of the maze. Each task has a set of low-level actions called motor primitives that make up the playing possibilities for the agent (e.g., straight shot, defend goal, and roll ball to corner). In the first stage, a nearest neighbor classifier is used to select the action to be performed. By observing the state of the game, the classifier searches for the most similar instances in the demonstrations provided by the human expert and retrieves the primitive selected by the human at that point. The next step is to define the goal of the selected action, for example, the velocity of the ball or the position of the puck when the primitive is completed. The goal is then used in a regression model to find the parameters of the action that would optimize the desired goal. The goal is derived from the k nearest neighbor demonstrations found in the previous step. The goals in those demonstrations are input in a local weighted regression model to perform the primitive. In a similar fashion, BIB002 use a classifier to make decisions in a sorting task consisting of the following macro actions (wait, sort left, sort right, and pass). Each macro action entails temporal motor actions such as picking up a ball and moving and placing the ball. Table I shows a list of methods used for direct imitation in the literature, along with the year, the domain in which imitation learning was used, and whether additional learning methods were used to improve learning from demonstrations. Popular applications in robotics are given their own category, such as navigation or batting (which are applications where a robot limb moves to make contact with an object, such as table tennis). More diverse or generic tasks are listed as robotics. The table shows that robotics and games are popular domains for imitation learning. They cover a wide variety of applications where an intelligent agent acts in the real world and in simulated environments, respectively. Robotics is an attractive domain for AI research due to the huge potential in applications that can take advantage of sensing and motor control in the physical world, while video games can be attractive because they alleviate many challenges such as data capturing and sensor error, and thus allow development of new complex learning methods in a controlled and easily reproducible environment. Moreover, games have built-in scoring measures that can facilitate evaluation and even designing the learning process in reinforcement learning approaches.