Reinforcement learning in robotics: A survey <s> Meta-actions <s> Modular row housing used to construct row housing units of one story or more comprising a plurality of modular units, each having a height of one story and a width corresponding to one-half of the finished width of the row housing unit, each modular unit comprising a party wall along its length, front and rear walls, a floor and a ceiling with the front and rear walls having a width corresponding to one-half of the finished width, each modular unit having an opening at the face opposite to the party wall, said opening being defined by free edges, two of said modular units being assembled together to form a single story with the free edges of each modular unit being held together in abuttment by a sealing material to close off the opening of each modular unit, thereby to provide the first story or a subsequent story of the row housing unit. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Meta-actions <s> This paper discusses why traditional reinforcement learning methods, and algorithms applied to those models, result in poor performance in situated domains characterized by multiple goals, noisy state, and inconsistent reinforcement. We propose a methodology for designing reinforcement functions that take advantage of implicit domain knowledge in order to accelerate learning in such domains. The methodology involves the use of heterogeneous reinforcement functions and progress estimators, and applies to learning in domains with a single agent or with multiple agents. The methodology is experimentally validated on a group of mobile robots learning a foraging task. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Meta-actions <s> The behaviour of reinforcement learning (RL) algorithms is best understood in completely observable, finite state- and action-space, discrete-time controlled Markov-chains. Robot-learning domains, on the other hand, are inherently infinite both in time and space, and moreover they are only partially observable. In this article we suggest a systematic design method whose motivation comes from the desire to transform the task-to-be-solved into a finite-state, discrete-time, “approximately” Markovian task, which is completely observable too. The key idea is to break up the problem into subtasks and design controllers for each of the subtasks. Then operating conditions are attached to the controllers (together the controllers and their operating conditions which are called modules) and possible additional features are designed to facilitate observability. A new discrete time-counter is introduced at the “module-level” that clicks only when a change in the value of one of the features is observed. The approach was tried out on a real-life robot. Several RL algorithms were compared and it was found that a model-based approach worked best. The learnt switching strategy performed equally well as a handcrafted version. Moreover, the learnt strategy seemed to exploit certain properties of the environment which could not have been seen in advance, which predicted the promising possibility that a learnt controller might overperform a handcrafted switching strategy in the future. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Meta-actions <s> Abstract This paper addresses adaptive control architectures for systems that respond autonomously to changing tasks. Such systems often have many sensory and motor alternatives and behavior drawn from these produces varying quality solutions. The objective is then to ground behavior in control laws which, combined with resources, enumerate closed-loop behavioral alternatives. Use of such controllers leads to analyzable and predictable composite systems, permitting the construction of abstract behavioral models. Here, discrete event system and reinforcement learning techniques are employed to constrain the behavioral alternatives and to synthesize behavior on-line. To illustrate this, a quadruped robot learning a turning gait subject to safety and kinematic constraints is presented. <s> BIB004 </s> Reinforcement learning in robotics: A survey <s> Meta-actions <s> This paper describes a formulation of reinforcement learning that enables learning in noisy, dynamic environments such as in the complex concurrent multi-robot learning domain. The methodology involves minimizing the learning space through the use of behaviors and conditions, and dealing with the credit assignment problem through shaped reinforcement in the form of heterogeneous reinforcement functions and progress estimators. We experimentally validate the approach on a group of four mobile robots learning a foraging task. <s> BIB005 </s> Reinforcement learning in robotics: A survey <s> Meta-actions <s> This paper presents a method of vision-based reinforcement learning by which a robot learns to shoot a ball into a goal. We discuss several issues in applying the reinforcement learning method to a real robot with vision sensor by which the robot can obtain information about the changes in an environment. First, we construct a state space in terms of size, position, and orientation of a ball and a goal in an image, and an action space is designed in terms of the action commands to be sent to the left and right motors of a mobile robot. This causes a “state-action deviation” problem in constructing the state and action spaces that reflect the outputs from physical sensors and actuators, respectively. To deal with this issue, an action set is constructed in a way that one action consists of a series of the same action primitive which is successively executed until the current state changes. Next, to speed up the learning time, a mechanism of Learning from Easy Missions (or LEM) is implemented. LEM reduces the learning time from exponential to almost linear order in the size of the state space. The results of computer simulations and real robot experiments are given. <s> BIB006 </s> Reinforcement learning in robotics: A survey <s> Meta-actions <s> Humans frequently engage in activities for their own sake rather than as a step towards solving a specific task. During such behavior, which psychologists refer to as being intrinsically motivated, we often develop skills that allow us to exercise mastery over our environment. Reference [7] have recently proposed an algorithm for intrinsically motivated reinforcement learning (IMRL) aimed at constructing hierarchies of skills through self-motivated interaction of an agent with its environment. While they were able to successfully demonstrate the utility of IMRL in simulation, we present the first realization of this approach on a real robot. To this end, we implemented a control architecture for the Sony-AIBO robot that extends the IMRL algorithm to this platform. Through experiments, we examine whether the Aibo is indeed able to learn useful skill hierarchies. <s> BIB007 </s> Reinforcement learning in robotics: A survey <s> Meta-actions <s> In the control-based approach to robotics, complex behavior is created by sequencing and combining control primitives. While it is desirable for the robot to autonomously learn the correct control sequence, searching through the large number of potential solutions can be time consuming. This paper constrains this search to variations of a generalized solution encoded in a framework known as an action schema. A new algorithm, schema structured learning, is proposed that repeatedly executes variations of the generalized solution in search of instantiations that satisfy action schema objectives. This approach is tested in a grasping task where Dexter, the UMass humanoid robot, learns which reaching and grasping controllers maximize the probability of grasp success. <s> BIB008 </s> Reinforcement learning in robotics: A survey <s> Meta-actions <s> General-purpose autonomous robots need to have the ability to sequence and adapt the available sensorimotor knowledge, which is often given in the form of movement primitives. In order to solve a given task in situations that were not considered during the initial learning, it is necessary to adapt trajectories contained in the library of primitive motions to new situations. In this paper we explore how to apply reinforcement learning to modify the subgoals of primitive movements involved in the given task. As the underlying sensorimotor representation we selected nonlinear dynamic systems, which provide a powerful machinery for the modification of motion trajectories. We propose a new formulation for dynamic systems, which ensures that consecutive primitive movements can be splined together in a continuous way (up to second order derivatives). <s> BIB009
Automatic construction of metaactions (and the closely related concept of options) has fascinated reinforcement learning researchers and there are various examples in the literature. The idea is to have more intelligent actions that are composed of a sequence of movements and that in themselves achieve a simple task. A simple example would be to have a meta-action "move forward 5 m". A lower-level system takes care of accelerating, stopping, and correcting errors. For example, in BIB006 , the state and action sets are constructed in a way that repeated action primitives lead to a change in the state to overcome problems associated with the discretization. Q-learning and dynamic programming based approaches have been compared in a pick-n-place task BIB003 using modules. BIB004 used a set of controllers with associated predicate states as a basis for learning turning gates with a quadruped.  used a policysearch approach to learn a small set of parameters that controls the transition between a walking and a capturing meta-action in a RoboCup scenario. A task of transporting a ball with a dog robot BIB007 can be learned with semi-automatically discovered options. Using only the sub-goals of primitive motions, a humanoid robot can learn a pouring task BIB009 . Other examples include foraging BIB005 and cooperative tasks BIB002 with multiple robots, grasping with restricted search spaces BIB008 , and mobile robot navigation BIB001 . If the meta-actions are not fixed in advance, but rather learned Table 3 . Different methods of making robot reinforcement learning tractable by employing a suitable representation.