Imitation Learning: A Survey of Learning Methods <s> Transfer Learning <s> We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Transfer Learning <s> A general game player is an agent capable of taking as input a description of a game's rules in a formal language and proceeding to play without any subsequent human input. To do well, an agent should learn from experience with past games and transfer the learned knowledge to new problems. We introduce a graph-based method for identifying previously encountered games and prove its robustness formally. We then describe how the same basic approach can be used to identify similar but non-identical games. We apply this technique to automate domain mapping for value function transfer and speed up reinforcement learning on variants of previously played games. Our approach is fully implemented with empirical results in the general game playing system. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Transfer Learning <s> Transfer learning has proven to be a wildly successful approach for speeding up reinforcement learning. Techniques often use low-level information obtained in the source task to achieve successful transfer in the target task. Yet, a most general transfer approach can only assume access to the output of the learning algorithm in the source task, i.e. the learned policy, enabling transfer irrespective of the learning algorithm used in the source task. We advance the state-of-the-art by using a reward shaping approach to policy transfer. One of the advantages in following such an approach, is that it firmly grounds policy transfer in an actively developing body of theoretical research on reward shaping. Experiments in Mountain Car, Cart Pole and Mario demonstrate the practical usefulness of the approach. <s> BIB003
Transfer learning is a machine-learning paradigm where knowledge of a task or a domain is used to enhance learning of another task. A learner can acquire various forms of knowledge about a task from another agent such as useful feature representations or parameters for the learning model. Transfer learning is relevant to imitation learning and robotic applications because acquiring samples is difficult and costly. Utilizing knowledge of a task that we already invested in to learn can be efficient and effective. A policy learned in one task can be used to advise (train) a learner on another task that carries some similarities. In BIB001 , this approach is implemented on two robocup soccer simulator tasks; the first is to keep the ball from the other team, and the second is to score a goal. It is obvious that skills learned to perform the first task can be of use in the latter. In this case, advice is formulated as a rule concerning the state and one or more action. To create advice, the policy for the first task is learned using reinforcement learning. The learned policy is then mapped by a user (to avoid discrepancies in state or action spaces) into the form of advice that is used to initiate the policy for the second task. After receiving advice, the learner continues to refine the policy through reinforcement learning and can modify or ignore the given advice if it proves through experience to be inaccurate or irrelevant. Often in transfer learning, human input is needed to map the knowledge from one domain to another; however, in some cases, the mapping procedure can be automated  . For example, in BIB002 , a mapping function for general game playing is presented. The function automatically maps between different domains to learn from previous experience. The agent is able to identify previously played games relevant to the current task. The agent may have played the same game before or a similar one and is able to select an appropriate source task to learn from without it being explicitly designated. Experiments show that the transfer learning approach speeds up the process of learning the game via reinforcement learning (compared to learning from scratch) and achieves a better performance after the learning iterations are complete. The results also suggest that the advantage of using transfer learning is correlated with the number of training instances transferred from the source tasks. Even if the agent encounters negative transfer  , for example, from overfitting to the source task, it can recover by learning through experience and rectifying its model in the current task to converge in an appropriate time BIB002 . BIB003 combine reward shaping and transfer learning to learn a variety of benchmark tasks. Since reward shaping relies on prior knowledge to influence the reward function, transfer learning can take advantage of a policy learned for one task and perform reward shaping for a similar task. In BIB003 , transfer learning is applied from a simple version of the problem to a more complex one (e.g., 2D to 3D mountain car and a Mario game without enemies to a game with enemies).