In this paper we focus on developing a control algorithm for multi-terrain tracked robots with flippers using a reinforcement learning (RL) approach. The work is based on the deep deterministic policy gradient (DDPG) algorithm, proven to be very successful in simple simulation environments. The algorithm works in an end-to-end fashion in order to control the continuous position of the flippers. This end-to-end approach makes it easy to apply the controller to a wide array of circumstances, but the huge flexibility comes to the cost of an increased difficulty of solution. The complexity of the task is enlarged even more by the fact that real multi-terrain robots move in partially observable environments. Notwithstanding these complications, being able to smoothly control a multi-terrain robot can produce huge benefits in impaired people daily lives or in search and rescue situations.
Most learning algorithms are not invariant to the scale of the function that is being approximated. We propose to adaptively normalize the targets used in learning. This is useful in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the policy of behavior. Our main motivation is prior work on learning to play Atari games, where the rewards were all clipped to a predetermined range. This clipping facilitates learning across many different games with a single learning algorithm, but a clipped reward function can result in qualitatively different behavior. Using the adaptive normalization we can remove this domain-specific heuristic without diminishing overall performance. Received Feb 6, 2012 Revised Mar 12, 2012 Accepted Mar 21, 2012 This paper proposes a behavior-switching control strategy of an evolutionary robotics based on Artificial Neural Network (ANN) and Genetic Algorithms (GA). This method is able not only to construct the reinforcement learning models for autonomous robots and evolutionary robot modules that control behaviors and reinforcement learning environments, and but also to perform the behavior-switching control and obstacle avoidance of an evolutionary robotics (ER) in time-varying environments with static and moving obstacles by combining ANN and GA. The experimental results on the basic behaviors and behavior-switching control have demonstrated that our method can perform the decision-making strategy and parameters set optimization of FNN and GA by learning and can escape successfully from the trap of a local minima and avoid “motion deadlock” status of humanoid soccer robotics agents, and reduce the oscillation of the planned trajectory between the multiple obstacles by crossover and mutation. Some results of the proposed algorithm have been successfully applied to our simulation humanoid robotics soccer team CIT3D which won the 1st prize of RoboCup Championship and China Open 2010 (July 2010) and the 2nd place of the official Robo Cup World Championship (2011) on 5-11 July, 2011 in Istanbul, Turkey. As compared with the conventional behavior network and the adaptive behavior method, the genetic encoding complexity of our algorithm is simplified, and the network performance and the convergence rate ρ have been greatly improved. Keyword: