Reinforcement learning in robotics: A survey <s> Hand-crafted discretization A variety of authors <s> In the Real-Time Learning Laboratory at GTE Laboratories, machine learning algorithms are being implemented on hardware testbeds. A modified connectionist actor-critic system has been applied to a ball balancing task. The system learns to balance a ball on a beam in less than 5 min and maintains the balance. A ball can roll along a few inches of a track on a flat metal beam, which an electric motor can rotate. A computer learning system running on a PC senses the position of the ball and the angular position of the beam. The system learns to prevent the ball from reaching either end of the beam. The system has shown to be robust through sensor noise and mechanical changes; it has also generated many interesting questions for future research. > <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Hand-crafted discretization A variety of authors <s> Since sensors have limited range and coverage, mobile robots often have to make decisions on where to point their sensors. A good sensing strategy allows a robot to collect information that is useful for its tasks. Most existing solutions to this active sensing problem choose the direction that maximally reduces the uncertainty in a single state variable. In more complex problem domains, however, uncertainties exist in multiple state variables, and they affect the performance of the robot in different ways. The robot thus needs to have more sophisticated sensing strategies in order to decide which uncertainties to reduce, and to make the correct trade-offs. In this work, we apply a least squares reinforcement learning method to solve this problem. We implemented and tested the learning approach in the RoboCup domain, where the robot attempts to reach a ball and accurately kick it into the goal. We present experimental results that suggest our approach is able to learn highly effective sensing strategies. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Hand-crafted discretization A variety of authors <s> Recently, the aspect of visual perception has been explored in the context of Gibson's concept of affordances [1] in various ways. We focus in this work on the importance of developmental learning and the perceptual cueing for an agent's anticipation of opportunities for interaction, in extension to functional views on visual feature representations. The concept for the incremental learning of abstract from basic affordances is presented in relation to learning of complex affordance features. In addition, the work proposes that the originally defined representational concept for the perception of affordances - in terms of using either motion or 3D cues - should be generalized towards using arbitrary visual feature representations. We demonstrate the learning of causal relations between visual cues and associated anticipated interactions by reinforcement learning of predictive perceptual states. We pursue a recently presented framework for cueing and recognition of affordance-based visual entities that obviously plays an important role in robot control architectures, in analogy to human perception. We experimentally verify the concept within a real world robot scenario by learning predictive visual cues using reinforcement signals, proving that features were selected for their relevance in predicting opportunities for interaction. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Hand-crafted discretization A variety of authors <s> We present a little crawling robot with a two DOF arm that learns to move forward within about 15 seconds in real time. Due to its small size and weight the robot is ideally suited for classroom demonstrations as well as for talks to the public. Students who want to practice their knowledge about reinforcement learning and value iteration can use a wireless connection to a PC and monitor the internal state of the robot such as the value function or the reward table. Due to its adaptivity, depending on the surface properties of the underground the robot may surprise its audience with unexpected but efficient walking policies. The GUI is open source and the robot hardware is available as a kit from the authors. <s> BIB004 </s> Reinforcement learning in robotics: A survey <s> Hand-crafted discretization A variety of authors <s> In the paper we evaluate two learning methods applied to the ball-in-a-cup game. The first approach is based on imitation learning. The captured trajectory was encoded with Dynamic motion primitives (DMP). The DMP approach allows simple adaptation of the demonstrated trajectory to the robot dynamics. In the second approach, we use reinforcement learning, which allows learning without any previous knowledge of the system or the environment. In contrast to the majority of the previous attempts, we used SASRA learning algorithm. Experimental results for both cases were performed on Mitsubishi PA10 robot arm. <s> BIB005
have manually developed discretizations so that basic tasks can be learned on real robots. For low-dimensional tasks, we can generate discretizations straightforwardly by splitting each dimension into a number of regions. The main challenge is to find the right number of regions for each dimension that allows the system to achieve a good final performance while still learning quickly. Example applications include balancing a ball on a beam BIB001 , one-DoF ball in a cup BIB005 , two-DoF crawling motions BIB004 , and gait patterns for four-legged walking  . Much more human experience is needed for more complex tasks. For example, in a basic navigation task with noisy sensors  , only some combinations of binary state or action indicators are useful (e.g. you can drive left and forward at the same time, but not backward and forward). The state-space can also be based on vastly different features, such as positions, shapes, and colors, when learning object affordances BIB003 where both the discrete sets and the mapping from sensor values to the discrete values need to be crafted. BIB002 used a mixed discrete and continuous representation of the state-space to learn active sensing strategies in a RoboCup scenario. They first discretize the state-space along the dimension with the strongest nonlinear influence on the value function and subsequently employ a linear value-function approximation (Section 4.2) for each of the regions.