Imitation Learning: A Survey of Learning Methods <s> MULTIAGENT IMITATION <s> Imitation is actively being studied as an effective means of learning in multi-agent environments. It allows an agent to learn how to act well (perhaps optimally) by passively observing the actions of cooperative teachers or other more experienced agents its environment. We propose a straightforward imitation mechanism called model extraction that can be integrated easily into standard model-based reinforcement learning algorithms. Roughly, by observing a mentor with similar capabilities, an agent can extract information about its own capabilities in unvisited parts of state space. The extracted information can accelerate learning dramatically. We illustrate the benefits of model extraction by integrating it with prioritized sweeping, and demonstrating improved performance and convergence through observation of single and multiple mentors. Though we make some stringent assumptions regarding observability, possible interactions and common abilities, we briefly comment on extensions of the model that relax these. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> MULTIAGENT IMITATION <s> We survey the recent work in AI on multi-agent reinforcement learning (that is, learning in stochastic games). We then argue that, while exciting, this work is flawed. The fundamental flaw is unclarity about the problem or problems being addressed. After tracing a representative sample of the recent literature, we identify four well-defined problems in multi-agent reinforcement learning, single out the problem that in our view is most suitable for AI, and make some remarks about how we believe progress is to be made on this problem. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> MULTIAGENT IMITATION <s> Humanoid robots working alongside humans in everyday environments is a long standing goal of the robotics community. To achieve this goal, methods for developing new robot behaviors that are intuitive and accessible to non-programmers are required. In this paper, we present a demonstration-based method for teaching distributed autonomous robots to coordinate their actions and perform collaborative multi-robot tasks. Within the presented framework, each robot learns an individual policy from teacher demonstrations using a confidence-based algorithm. Based on this learning approach, we contribute three techniques for teaching multi-robot coordination using different information sharing strategies. We evaluate and compare these approaches by teaching two Sony QRIO humanoid robots to perform three collaborative ball sorting tasks. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> MULTIAGENT IMITATION <s> Developing coordination among multiple agents and enabling them to exhibit teamwork is a challenging yet exciting task that can benefit many of the complex real-life problems. This research uses imitation to learn collaborative strategies for a team of agents. Imitation based learning involves learning from an expert by observing him/her demonstrating a task and then replicating it. The key idea is to involve multiple human experts during demonstration to teach autonomous agents how to work in coordination. The effectiveness of the proposed methodology has been assessed in a goal defending scenario of the RoboCup Soccer Simulation 3D league. The process involves multiple human demonstrators controlling soccer agents via game controllers and demonstrating them how to play soccer in coordination. The data gathered during this phase is used as training data to learn a classification model which is later used by the soccer agents to make autonomous decisions during actual matches. Different performance evaluation metrics are derived to compare the performance of imitating agent with that of the human-driven agent and hand-coded (if-then-else rules) agent. <s> BIB004
Although creating autonomous multiagents has been thoroughly investigated through reinforcement learning BIB002  , it is not as extensively explored in imitation learning. Despite the lack of research, imitation learning and multiagent applications can be a good fit. Learning from demonstrations can be improved in multiagent environments as knowledge can be transferred between agents of similar objectives. On the other hand, imitation learning can be beneficial in tasks where agents need to interact in a manner that is realistic from a human's perspective. In the following, we present methods that incorporate imitation learning in multiple agents. In BIB001 , implicit imitation is used to improve a learner's RL model. A multiagent setting enables an agent to learn by observing other agents performing a similar task using similar actions to those possessed by the agent. The mentor agent provides demonstrations by performing the task to optimize its objectives, so there is no need for a designated teacher, and the actions of an agent are unknown to other agents. A learner can observe the state changes resulting from the actions of a mentor agent and accordingly refine its model. This premise is useful for real applications where multiple agents act in the same environment. However, this work assumes that the agents are noninteracting; that is, the consequences of one agent's actions are independent of other agents. Implicit imitation is closely related to transfer learning as a learner acquires the knowledge learned previously by a different learner. This application corresponds to a transductive transfer learning setting where the task is the same but the domains of the mentor and learner are different  ]. An interesting aspect of this approach is that an agent can learn different skills from different mentors. In one experiment, two mentors are acting to achieve different goals; the learner uses observations from both agents to learn a more difficult task. Note, however, that the tasks were designed so that a combination of the policies used by the mentors form an optimal policy for the learner's problem BIB001 . This can be considered an example of hierarchical transfer learning, where learning solutions to multiple problems can help achieve more complex tasks  . However, in this case, the knowledge of the tasks is learned through observations and imitation. Multiagent settings add complexity to the imitation problem in a number of ways. The learner's state space can be significantly expanded to include the status of other agents, as observing the actions of other agents affects its decision. The reward function is also affected if multiple agents compete or collaborate toward the same goal. The complexity of reward functions can increase even if the agents do not share the same goal. In a competitive setting, in addition to maximizing its own reward, an agent aims to minimize its opponent's reward. In a cooperative setting, the total reward of the team might be taken into consideration. These new complexities can be incorporated in the learning process at different levels. For example, BIB001 exploit the presence of multiple agents to learn from observation; however, robots are noninteracting and act independently of each other. So state space and reward function are not affected. In "keep-away" in robot soccer, the robots collaborate as a team to keep the ball from the other team. However, the whole team shares a Q policy, because they all perform the same task. When a new action is performed, the shared Q policy of all players is updated and the same reward is applied to the entire team, while in BIB004 , agents learn different roles that complement each other to maximize the common goal. In a defensive task, one agent tries to recover the ball from the attacking opponents, while the other falls back to act as a goal keeper. The roles are interchangeable, so each agent learns both skills as well as when to assume one of the two roles. In DAmbrosio and Stanley [2013] , a team of agents is treated as a pattern of policies rather than individual agents. That is a pattern that connects agents in a team that perform complementary roles. This method enables a team of agents to be scaled significantly after training without requiring retraining of different agents for similar roles. Multiagent learning from demonstration can, however, introduce new challenges in terms of acquiring demonstrations. In BIB003 , where active learning is employed in a multirobot environment, the human expert is required to interact simultaneously with multiple robots. A system is then developed to divide the expert's time among the learners based on their need by attracting the expert's attention through audio-visual cues in order to query information.