Reinforcement learning in robotics: A survey <s> Directing exploration with prior knowledge <s> This dissertation addresses the problem of designing algorithms for learning in embedded systems. This problem differs from the traditional supervised learning problem. An agent, finding itself in a particular input situation must generate an action. It then receives a reinforcement value from the environment, indicating how valuable the current state of the environment is for the agent. The agent cannot, however, deduce the reinforcement value that would have resulted from executing any of its other actions. A number of algorithms for learning action strategies from reinforcement values are presented and compared empirically with existing reinforcement-learning algorithms. ::: The interval-estimation algorithm uses the statistical notion of confidence intervals to guide its generation of actions in the world, trading off acting to gain information against acting to gain reinforcement. It performs well in simple domains but does not exhibit any generalization and is computationally complex. ::: The cascade algorithm is a structural credit-assignment method that allows an action strategy with many output bits to be learned by a collection of reinforcement-learning modules that learn Boolean functions. This method represents an improvement in computational complexity and often in learning rate. ::: Two algorithms for learning Boolean functions in k-DNF are described. Both are based on Valiant's algorithm for learning such functions from input-output instances. The first uses Sutton's techniques for linear association and reinforcement comparison, while the second uses techniques from the interval estimation algorithm. They both perform well and have tractable complexity. ::: A generate-and-test reinforcement-learning algorithm is presented. It allows symbolic representations of Boolean functions to be constructed incrementally and tested in the environment. It is highly parametrized and can be tuned to learn a broad range of function classes. Low-complexity functions can be learned very efficiently even in the presence of large numbers of irrelevant input bits. This algorithm is extended to construct simple sequential networks using a set-reset operator, which allows the agent to learn action strategies with state. ::: These algorithms, in addition to being studied in simulation, were implemented and tested on a physical mobile robot. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Directing exploration with prior knowledge <s> Autonomous approaching objects for an arm-hand robot is a very difficult problem because the possible arm-hand configurations are numerous. In this paper, we propose a modified reinforcement learning algorithm for a multifingerd hand approaching target objects. The proposed approach integrates the heuristic search information with the learning system, and solves the problem of how an arm-hand robot approaches objects before grasping. In addition, this method also overcomes the problem of time consuming of traditional reinforcement learning in the initial learning phase. The algorithm is applied to an arm-hand robot to approach objects before grasping, which can enable the robot to learn approaching skill by trial-and-error and plan its path by itself. The experimental results demonstrate the effectiveness of the proposed algorithm. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Directing exploration with prior knowledge <s> Grasping an object is a task that inherently needs to be treated in a hybrid fashion. The system must decide both where and how to grasp the object. While selecting where to grasp requires learning about the object as a whole, the execution only needs to reactively adapt to the context close to the grasp's location. We propose a hierarchical controller that reflects the structure of these two sub-problems, and attempts to learn solutions that work for both. A hybrid architecture is employed by the controller to make use of various machine learning methods that can cope with the large amount of uncertainty inherent to the task. The controller's upper level selects where to grasp the object using a reinforcement learner, while the lower level comprises an imitation learner and a vision-based reactive controller to determine appropriate grasping motions. The resulting system is able to quickly learn good grasps of a novel object in an unstructured environment, by executing smooth reaching motions and preshaping the hand depending on the object's geometry. The system was evaluated both in simulation and on a real robot. <s> BIB003
As discussed in Section 2.1, balancing exploration and exploitation is an important consideration. Task knowledge can be employed to guide to robots curiosity to focus on regions that are novel and promising at the same time. For example, a mobile robot learns to direct attention by employing a modified Q-learning approach using novelty  . Using "corrected truncated returns" and taking into account the estimator variance, a six-legged robot employed with stepping reflexes can learn to walk  . Offline search can be used to guide Q-learning during a grasping task BIB002 . Using upper confidence bounds BIB001 to direct exploration into regions with potentially high rewards, grasping can be learned efficiently BIB003 ).