Imitation Learning: A Survey of Learning Methods <s> Apprenticeship Learning <s> We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Apprenticeship Learning <s> Recent research has shown the benefit of framing problems of imitation learning as solutions to Markov Decision Problems. This approach reduces learning to the problem of recovering a utility function that makes the behavior induced by a near-optimal policy closely mimic demonstrated behavior. In this work, we develop a probabilistic approach based on the principle of maximum entropy. Our approach provides a well-defined, globally normalized distribution over decision sequences, while providing the same performance guarantees as existing methods. ::: ::: We develop our technique in the context of modeling real-world navigation and driving behaviors where collected data is inherently noisy and imperfect. Our probabilistic approach enables modeling of route preferences as well as a powerful new approach to inferring destinations and routes based on partial trajectories. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Apprenticeship Learning <s> This paper presents a general framework for exploiting the representational capacity of neural networks to approximate complex, nonlinear reward functions in the context of solving the inverse reinforcement learning (IRL) problem. We show in this context that the Maximum Entropy paradigm for IRL lends itself naturally to the efficient training of deep architectures. At test time, the approach leads to a computational complexity independent of the number of demonstrations, which makes it especially well-suited for applications in life-long learning scenarios. Our approach achieves performance commensurate to the state-of-the-art on existing benchmarks while exceeding on an alternative benchmark based on highly varying reward structures. Finally, we extend the basic architecture - which is equivalent to a simplified subclass of Fully Convolutional Neural Networks (FCNNs) with width one - to include larger convolutions in order to eliminate dependency on precomputed spatial features and work on raw input representations. <s> BIB003
In many artificial intelligence applications such as games or complex robotic tasks, the success of an action is hard to quantify. In that case, the demonstrated samples can be used as a template for the desired performance. In BIB001 , apprenticeship learning (or inverse reinforcement learning) is proposed to improve a learned policy when no clear reward function is available, such as the task of driving. In such applications, the aim is to mimic the behavior of the human teachers under the assumption that the teacher is optimizing an unknown function. Definition 15. Inverse reinforcement learning (IRL) uses the training samples to learn the reward function being optimized by the expert and use it to improve the trained model. Thus, IRL obtains performance similar to that of the expert. With no reward function, the agent is modeled as an MDP/R (S,A,T). Instead, the policy is modeled after feature expectations μ E derived from experts' demonstrations. Given n trajectories {s , the empirical estimate for the feature expectation of the expert's policy where γ is a discount factor and φ(s t ) is the feature vector at time t of demonstration i. The goal of the RL algorithm is to find a policyπ such that ||μ(π) − μ E || 2 ≤ , where μ(π ) is the expectation of the policy BIB001 . BIB002 employ a maximum entropy approach to IRL to alleviate ambiguity. Ambiguity arises in IRL tasks since many reward functions can be optimized by the same policy. This poses a problem when learning the reward function, especially when presented with imperfect demonstrations. The proposed method is demonstrated on a task of learning driver route choices where the demonstrations may be suboptimal and nondeterministic. This approach is extended to a deep learning framework in BIB003 . Maximum entropy objective functions enable straightforward learning of the network weights, and thus the use of deep networks trained with stochastic gradient descent BIB003 . The deep architecture is further extended to learn the features via convolution layers instead of using pre-extracted features. This is an important step in the route to automate the learning process. One of the main challenges in reinforcement learning through trial and error is the requirement of human knowledge in designing the feature representations and reward functions . By using deep learning to automatically learn feature representations and using IRL to infer reward functions from demonstrations, the need for human input and design is minimized. The inverse reinforcement learning paradigm provides an advantage over other forms of learning from demonstrations in that the cost function of the task is decoupled from the environment. Since the objective of the demonstrations is learned rather than demonstrations themselves, the demonstrator and learner do not need to have the exact skeleton or surroundings, thus alleviating challenges such as the correspondence problem. Therefore, it is easier to provide demonstrations that are generic and not tailor-made for a specific robot or environment. In addition, IRL can be employed rather than traditional RL even if a reward function exists (given that demonstrations are available). For example, in , apprenticeship learning is used to derive a reward function from expert demonstrations in a Mario game. While the goals in a game such as Mario can be predefined (such as the score from killing enemies and collecting coins or the time to complete the level), it is not known how an expert user prioritizes these goals. So in an effort to mimic human behavior, a reward function extracted from demonstrations is favored to a manually designed reward function.