Planning problems in partially observable environments cannot be solved directly with convolutional networks and require some form of memory. But, even memory networks with sophisticated addressing schemes are unable to learn intelligent reasoning satisfactorily due to the complexity of simultaneously learning to access memory and plan. To mitigate these challenges we propose the Memory Augmented Control Network (MACN). The network splits planning into a hierarchical process. At a lower level, it learns to plan in a locally observed space. At a higher level, it uses a collection of policies computed on locally observed spaces to learn an optimal plan in the global environment it is operating in. The performance of the network is evaluated on path planning tasks in environments in the presence of simple and complex obstacles and in addition, is tested for its ability to generalize to new environments not seen in the training set. A planning task in a partially observable environment involves two steps: inferring the environment structure from local observation and acting based on the current environment estimate. In the past, such perception-action loops have been learned using supervised learning with deep networks as well as deep reinforcement learning BID3 , BID1 , . Popular approaches in this spirit are often end-to-end (i.e. mapping sensor readings directly to motion commands) and manage to solve problems in which the underlying dynamics of the environment or the agent are too complex to model. Approaches to learn end-to-end perception-action loops have been extended to complex reinforcement learning tasks such as learning how to play Atari games (Mnih et al., 2013a) , as well as to imitation learning tasks like controlling a robot arm BID12 .Purely convolutional architectures (CNNs) perform poorly when applied to planning problems due to the reactive nature of the policies learned by them BID21 , BID4 . The complexity of this problem is compounded when the environment is only partially observable as is the case with most real world tasks. In planning problems , when using a function approximator such as a convolutional neural network, the optimal actions are dependent on an internal state. If one wishes to use a state-less network (such as a CNN) to obtain the optimal action, the input for the network should be the whole history of observations and actions. Since this does not scale well, we need a network that has an internal state such as a recurrent neural network or a memory network. BID20 showed that when learning how to plan in partially observable environments, it becomes necessary to use memory to retain information about states visited in the past. Using recurrent networks to store past information and learn optimal control has been explored before in BID11 . While BID14 have shown that recurrent networks are Turing complete and are hence capable of generating any arbitrary sequence in theory, this does not always translate into practice. Recent advances in memory augmented networks have shown that it is beneficial to use external memory with read and write operators that can be learned by a neural network over recurrent neural networks BID5 , BID6 . Specifically, we are interested in the Differentiable Neural Computer (DNC) BID6 which uses an external memory and a network controller to learn how to read, write and access locations in the external memory. The DNC is structured such that computation and memory operations are separated from each other. Such a memory network can in principle be plugged into the convolutional architectures described above, and be trained end to end since the read and write operations are differentiable. However, as we show in our work, directly using such a memory scheme with CNNs performs poorly for partially observable planning problems and also does not generalize well to new environments.To address the aforementioned challenges we propose the Memory Augmented Control Network (MACN), a novel architecture specifically designed to learn how to plan in partially observable environments under sparse rewards.1 Environments with sparse rewards are harder to navigate since there is no immediate feedback. The intuition behind this architecture is that planning problem can be split into two levels of hierarchy. At a lower level, a planning module computes optimal policies using a feature rich representation of the locally observed environment. This local policy along with a sparse feature representation of the partially observed environment is part of the optimal solution in the global environment. Thus, the key to our approach is using a planning module to output a local policy which is used to augment the neural memory to produce an optimal policy for the global environment. Our work builds on the idea of introducing options for planning and knowledge representation while learning control policies in MDPs BID16 . The ability of the proposed model is evaluated by its ability to learn policies (continuous and discrete) when trained in environments with the presence of simple and complex obstacles. Further, the model is evaluated on its ability to generalize to environments and situations not seen in the training set.The key contributions of this paper are:1. A new network architecture that uses a differentiable memory scheme to maintain an estimate of the environment geometry and a hierarchical planning scheme to learn how to plan paths to the goal. 2. Experimentation to analyze the ability of the architecture to learn how to plan and generalize in environments with high dimensional state and action spaces.2 METHODOLOGY Section 2.1 outlines notation and formally states the problem considered in this paper. Section 2.2 and 2.3 briefly cover the theory behind value iteration networks and memory augmented networks. Finally, in section 2.4 the intuition and the computation graph is explained for the practical implementation of the model. Planning in environments that are partially observable and have sparse rewards with deep learning has not received a lot of attention. Also, the ability of policies learned with deep RL to generalize to new environments is often not investigated. In this work we take a step toward designing architectures that compute optimal policies even when the rewards are sparse, and thoroughly investigate the generalization power of the learned policy. In addition we show our network is able to scale well to large dimensional spaces.The grid world experiments offer conclusive evidence about the ability of our network to learn how to plan in such environments. We address the concern of oversimplifying our environment to a 2D grid world by experimenting with planning in a graph with no constraint on the state space or the action space. We also show our model is capable of learning how to plan under continuous control. In the future, we intend to extend our policies trained in simulation to a real world platform such as a robot learning to plan in partially observable environments. Additionally, in our work we use simple perfect sensors and do not take into account sensor effects such as occlusion, noise which could aversely affect performance of the agent. This need for perfect labeling is currently a limitation of our work and as such cannot be applied directly to a scenario where a sensor cannot provide direct information about nearby states such as a RGB camera. We intend to explore this problem space in the future, where one might have to learn sensor models in addition to learning how to plan.
Memory Augmented Network to plan in partially observable environments. 
{"src": "scitldr-AIC_train", "paper_id": "HyfHgI6aW"}