We propose to deal with sequential processes where only partial observations are available by learning a latent representation space on which policies may be accurately learned.
Efficient approaches have been proposed to extract high-level representations using deep-learning @cite_1 but few studies have proposed extension to deal with sequential processes. A formal analysis has been proposed in @cite_6 . Models concerning partially observable sequential processes have been proposed in the context of controlling tasks problems. For example, @cite_5 and @cite_0 present models using recurrent neural networks (RNN) to learn a controller for a given task. In these approaches, informative representations are constructed by the RNN, but these representations are driven by the task to solve. Some unsupervised approaches have been recently proposed. In that case, a representation learning model is learned over the observations, without needing to define a reward function. The policy is learned afterward using these representations, by usually using classical RL algorithms. For instance, @cite_4 propose a model based on a recurrent auto-associative memory with history of arbitrary depth, while @cite_3 present an extension of RNN for unsupervised learning. In comparison to these models, our transductive approach is simultaenously based on unsupervised trajectories, and also allows us to choose which action to take even if observations are missing, by learning a dynamic model in the latent space.
@cite_4 @cite_1 @cite_6 @cite_3 @cite_0 @cite_5
130805539 2953130357 2002196558 2084920657
Traditional Reinforcement Learning methods are insufficient for AGIs who must be able to learn to deal with Partially Observable Markov Decision Processes. We investigate a novel method for dealing with this problem: standard RL techniques using as input the hidden layer output of a Sequential Constant-Size Compressor (SCSC). The SCSC takes the form of a sequential Recurrent Auto-Associative Memory, trained through standard back-propagation. Results illustrate the feasibility of this approach -- this system learns to deal with highdimensional visual observations (up to 640 pixels) in partially observable environments where there are long time lags (up to 12 steps) between relevant sensory information and necessary action. Numerous control and learning problems face the situation where sequences of high-dimensional highly dependent data are available, but no or little feedback is provided to the learner. To address this issue, we formulate the following problem. Given a series of observations X_0,...,X_n coming from a large (high-dimensional) space X, find a representation function f mapping X to a finite space Y such that the series f(X_0),...,f(X_n) preserve as much information as possible about the original time-series dependence in X_0,...,X_n. We show that, for stationary time series, the function f can be selected as the one maximizing the time-series information h_0(f(X))- h_ (f(X)) where h_0(f(X)) is the Shannon entropy of f(X_0) and h_ (f(X)) is the entropy rate of the time series f(X_0),...,f(X_n),... Implications for the problem of optimal control are presented. Neuroevolution, the artificial evolution of neural networks, has shown great promise on continuous reinforcement learning tasks that require memory. However, it is not yet directly applicable to realistic embedded agents using high-dimensional (e.g. raw video images) inputs, requiring very large networks. In this paper, neuroevolution is combined with an unsupervised sensory pre-processor or compressor that is trained on images generated from the environment by the population of evolving recurrent neural network controllers. The compressor not only reduces the input cardinality of the controllers, but also biases the search toward novel controllers by rewarding those controllers that discover images that it reconstructs poorly. The method is successfully demonstrated on a vision-based version of the well-known mountain car benchmark, where controllers receive only single high-dimensional visual images of the environment, from a third-person perspective, instead of the standard two-dimensional state vector which includes information about velocity. An online learning algorithm for reinforcement learning with continually running recurrent networks in nonstationary reactive environments is described. Various kinds of reinforcement are considered as special types of input to an agent living in the environment. The agent's only goal is to maximize the amount of reinforcement received over time. Supervised learning techniques for recurrent networks serve to construct a differentiable model of the environmental dynamics which includes a model of future reinforcement. This model is used for learning goal-directed behavior in an online fashion. The possibility of using the system for planning future action sequences is investigated and this approach is compared to approaches based on temporal difference methods. A connection to met alearning (learning how to learn) is noted