A Survey on Deep Learning Methods for Robot Vision <s> Deep Reinforcement Learning <s> A Markov Decision Process (MDP) is a natural framework for formulating sequential decision-making problems under uncertainty. In recent years, researchers have greatly advanced algorithms for learning and acting in MDPs. This article reviews such algorithms, beginning with well-known dynamic programming methods for solving MDPs such as policy iteration and value iteration, then describes approximate dynamic programming methods such as trajectory based value iteration, and finally moves to reinforcement learning methods such as Q-Learning, SARSA, and least-squares policy iteration. We describe algorithms in a unified framework, giving pseudocode together with memory and iteration complexity analysis for each. Empirical evaluations of these techniques with four representations across four domains, provide insight into how these algorithms perform with various feature sets in terms of running time and performance. <s> BIB001 </s> A Survey on Deep Learning Methods for Robot Vision <s> Deep Reinforcement Learning <s> The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. <s> BIB002 </s> A Survey on Deep Learning Methods for Robot Vision <s> Deep Reinforcement Learning <s> The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses ‘value networks’ to evaluate board positions and ‘policy networks’ to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of stateof-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away. <s> BIB003
Reinforcement Learning consists of a set of techniques designed to maximize the cumulative reward obtained by an agent when interacting with its environment. The agent has an internal state, and is able to observe its environment partially, to act on it, and to obtain an immediate reward. This problem can be modeled by using a Markov Decision Process (MDP), which is defined by: (i) are known, the optimal policy can be computed by using dynamic programming. However, when both are not known a priori, and can only be estimated by interacting with an environment, other approaches must be used. Classical reinforcement learning, in particular Q-learning, uses a discrete state space (usually by partitioning a continuous one) and learns the value function Q by performing several experimental trials, and exploiting the recursivity property for learning from successful ones. Also, approaches that use a linear parameterized function approximation, , with a number of parameters smaller than the number of states, has been used BIB001 . Deep reinforcement learning (DRL) is able to learn the optimal mapping, * Q , directly from states (represented as sets of consecutive raw observations) and actions related to the different outcomes for each action, by using a convolutional neural network, called a Deep Q Network (DQN). This technique has been applied successfully for agents to learn by playing several Atari games with only raw gray scale images and the scores in the game BIB002 , achieving human-level playing ability. As neural networks learn from new data, a technique named experience replay is used. It consists of selecting random samples from both old and new trials for training the network. Then, the learning process does not get biased by the last trials, and context information from old trials can be preserved and used BIB002 . Policy gradient methods and actor/critic algorithms have been proposed recently [195] for enabling end-to-end training and faster convergence and evaluation times with respect to DQNs. The use of supervised learning for policy networks as initialization for reinforcement learning has proven to be useful, and has enabled computers to defeat human professional players in the game of Go BIB003 . Deep reinforcement learning is a very hot research topic as it has the potential to solve real-world problems with a human-level performance, using end-to-end formulations.