Central Pattern Generators (CPGs) are biological neural circuits capable of producing coordinated rhythmic outputs in the absence of rhythmic input. As a result, they are responsible for most rhythmic motion in living organisms. This rhythmic control is broadly applicable to fields such as locomotive robotics and medical devices. In this paper, we explore the possibility of creating a self-sustaining CPG network for reinforcement learning that learns rhythmic motion more efficiently and across more general environments than the current multilayer perceptron (MLP) baseline models. Recent work introduces the Structured Control Net (SCN), which maintains linear and nonlinear modules for local and global control, respectively. Here, we show that time-sequence architectures such as Recurrent Neural Networks (RNNs) model CPGs effectively. Combining previous work with RNNs and SCNs, we introduce the Recurrent Control Net (RCN), which adds a linear component to the, RCNs match and exceed the performance of baseline MLPs and SCNs across all environment tasks. Our findings confirm existing intuitions for RNNs on reinforcement learning tasks, and demonstrate promise of SCN-like structures in reinforcement learning.
RNNs model time-sequences well by maintaining a hidden state as a function of priors @cite_14 . Previously, RNNs have been intensively used in natural language processing because sequence prediction is best modeled by including previous context. Therefore, to leverage the added context, RNNs has also been explored loosely in reinforcement learning for quadruped environments. However, RNNs have not been generalized to general locomotive tasks and still remain relatively specific to these quadruped based locomotion tasks @cite_18 @cite_13 . Instead of RNNs as policy networks, inclusion of linear-global dynamics by adopting simple RNN as a nonlinearity in SCNs have the potential to exceed existing SCN MLP Baselines (See Appendix for a comprehensive description of effective RNN architectures).
@cite_18 @cite_14 @cite_13
2154269964 1598796236
Aiming to real easy application in several quadruped robot platforms, this paper introduces a new method of modeling central pattern generators (CPG) to control quadruped locomotion. Not only can this new model generate all the primary gaits of quadrupeds stably with limit cycle effect, but it also has the ability of tuning the periodic outputs with arbitrary waveforms. The core idea is to combine strong points of two mathematical tools: Fourier series and Recurrent neural networks. In addition, a new biomimetic controller is also introduced using the proposed CPG model and several reflex modules. Finally, dynamic simulations are performed to validate the efficiency of the proposed controller. Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been dicult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades rst yielded and then made practical these powerful learning models. When appropriate, we reconcile conicting notation and nomenclature. Our goal is to provide a selfcontained explication of the state of the art together with a historical perspective and references to primary research.