Imitation Learning: A Survey of Learning Methods <s> Structured Predictions <s> Tying suture knots is a time-consuming task performed frequently during minimally invasive surgery (MIS). Automating this task could greatly reduce total surgery time for patients. Current solutions to this problem replay manually programmed trajectories, but a more general and robust approach is to use supervised machine learning to smooth surgeon-given training trajectories and generalize from them. Since knottying generally requires a controller with internal memory to distinguish between identical inputs that require different actions at different points along a trajectory, it would be impossible to teach the system using traditional feedforward neural nets or support vector machines. Instead we exploit more powerful, recurrent neural networks (RNNs) with adaptive internal states. Results obtained using LSTM RNNs trained by the recent Evolino algorithm show that this approach can significantly increase the efficiency of suture knot tying in MIS over preprogrammed control <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Structured Predictions <s> We present Searn, an algorithm for integrating search and learning to solve complex structured prediction problems such as those that occur in natural language, speech, computational biology, and vision. Searn is a meta-algorithm that transforms these complex problems into simple classification problems to which any binary classifier may be applied. Unlike current algorithms for structured learning that require decomposition of both the loss function and the feature functions over the predicted structure, Searn is able to learn prediction functions for any loss function and any class of features. Moreover, Searn comes with a strong, natural theoretical guarantee: good performance on the derived classification problems implies good performance on the structured prediction problem. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Structured Predictions <s> We address the problem of endowing a robot with the capability to learn a repertoire of actions using as little prior knowledge as possible. Taking a handwriting task as an example, we apply the deep learning paradigm to build a network which uses a high-level representation of digits to generate sequences of commands, directly fed to a low-level control loop. Discrete variables are used to discriminate different digits, while continuous variables parametrize each digit. We show that the proposed network is able to generalize learned actions to new contexts. The network is tested on trajectories recorded on the iCub humanoid robot. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> Structured Predictions <s> We study the problem of smooth imitation learning for online sequence prediction, where the goal is to train a policy that can smoothly imitate demonstrated behavior in a dynamic and continuous environment in response to online, sequential context input. Since the mapping from context to behavior is often complex, we take a learning reduction approach to reduce smooth imitation learning to a regression problem using complex function classes that are regularized to ensure smoothness. We present a learning meta-algorithm that achieves fast and stable convergence to a good policy. Our approach enjoys several attractive properties, including being fully deterministic, employing an adaptive learning rate that can provably yield larger policy improvements compared to previous approaches, and the ability to ensure stable convergence. Our empirical results demonstrate significant performance gains over previous approaches. <s> BIB004
In a similar spirit, DAGGER employs sample aggregation to generalize for unseen situations. However, the approach is fundamentally different. DAGGER formulates the imitation learning problem as a structured prediction problem inspired by BIB002 ; an action is regarded as a sequence of dependent predictions. Since each action is dependent on the previous state, an error leads to an unseen state from which the learner cannot recover, leading to compounded errors. DAGGER shows that it is both necessary and sufficient to aggregate samples that cover initial learning errors. Therefore, an iterative approach is proposed that uses an optimal policy to correct each step of the actions predicted using the current policy, thus creating new modified samples that are used to update the policy. As the algorithm iterates, the utilization of the optimal policy diminishes until only the learned policy is used as the final model. BIB004 propose an algorithm called SIMILE that mitigates the limitations of and BIB002 by producing a stationary policy that doesn't require data aggregation. SIMILE alleviates the need for an expert to provide the action at every stage of the trajectory by providing "virtual expert feedback" that controls the smoothness of the corrected trajectory and converges to the expert's actions. Considering past actions in the learning process is an important point in imitation learning as many applications rely on performing trajectories of dependent motion primitives. A generic method of incorporating memory in learning is using RNNs BIB003 . RNNs create a feedback loop among the hidden layers in order to consider the network's previous outputs and are therefore well suited for tasks with structured trajectories BIB001 . To conclude this section, Figure 5 shows a Venn diagram outlining the sources of data employed by different learning methods. An agent can learn from dedicated teacher demonstrations, by observing other agents' actions, or through trial and error. Active learning needs a dedicated oracle that can be queried for demonstrations, while other methods that utilize demonstrations can acquire them from a dedicated expert or by observing the required behavior from other agents. RL and optimization methods learn through trial and error and do not make use of demonstrations. Transfer learning uses experience from old tasks or knowledge from other agents to learn a new policy. Apprenticeship learning uses demonstrations from an expert or observation to learn a reward function. A policy that optimizes the reward function can then be learned through experience.