State Representation Learning for Control: An Overview <s> SRL approaches <s> Previous work has shown that the difficulties in learning deep generative or discriminative models can be overcome by an initial unsupervised learning step that maps inputs to useful intermediate representations. We introduce and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern. This approach can be used to train autoencoders, and these denoising autoencoders can be stacked to initialize deep architectures. The algorithm can be motivated from a manifold learning and information theoretic perspective or from a generative model perspective. Comparative experiments clearly show the surprising advantage of corrupting the input of autoencoders on a pattern classification benchmark suite. <s> BIB001 </s> State Representation Learning for Control: An Overview <s> SRL approaches <s> We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations. <s> BIB002 </s> State Representation Learning for Control: An Overview <s> SRL approaches <s> Robot learning is critically enabled by the availability of appropriate state representations. We propose a robotics-specific approach to learning such state representations. As robots accomplish tasks by interacting with the physical world, we can facilitate representation learning by considering the structure imposed by physics; this structure is reflected in the changes that occur in the world and in the way a robot can effect them. By exploiting this structure in learning, robots can obtain state representations consistent with the aspects of physics relevant to the learning task. We name this prior knowledge about the structure of interactions with the physical world robotic priors. We identify five robotic priors and explain how they can be used to learn pertinent state representations. We demonstrate the effectiveness of this approach in simulated and real robotic experiments with distracting moving objects. We show that our method extracts task-relevant state representations from high-dimensional observations, even in the presence of task-irrelevant distractions. We also show that the state representations learned by our method greatly improve generalization in reinforcement learning. <s> BIB003
Based on the previously defined notations, we can briefly summarize the common strategies used in state representation learning that are detailed in the next sections. In the following, θ represents the parameters optimized by minimizing the model's loss function. In most of the approaches we present, this model is implemented with a neural network. Figure 2: Auto-Encoder: reconstructing the observation. The error is computed between observation o t and its reconstructionô t . White components are input data and gray ones are output data. Figure 3: Forward Model: predicting next state s t+1 from s t and a t . The error is computed between predicted stateŝ t+1 and the actual next state s t+1 • Reconstructing the observation: learning the function φ (Eq. 1) so that it is possible to reconstruct the observation with a decoder φ −1 (Eq. 2) by minimizing the reconstruction error between the original observation and its predicted reconstruction. The reconstruction is learned under different constraints that give to s t specific characteristics (e.g., dimensionality constraints, local denoising criterion BIB002 , sparse encoding constraints BIB001 , etc.) (Fig. 2) . where θ φ and θ φ −1 are the parameters learned for the encoder and decoder, respectively. • Learning a forward model: A forward model predicts s t+1 from o t or s t and a t (Fig. 3) . In this context, we want to learn the mapping φ from o t to s t using the model that predicts s t+1 from o t . Hence the prediction work in two steps, first encoding from o t to s t then transition from s t toŝ t+1 . We can not compute any error on s t , however at t + 1 the model can learn from the error betweenŝ t+1 and s t+1 . The error is back propagated through the transition model and the encoding model. Consequently the methods allows to learn a model φ. Learning such a model makes it possible to impose structural constraints on the model for state representation learning. For example, the forward model can be constrained to be linear between s t and s t+1 , imposing that the system in the learned state space follows simple linear dynamics. Figure 5: Model with prior: The error is computed by applying loss functions on several states • Learning an inverse model: An inverse model predicts action a t given observations o t and o t+1 or states s t and s t+1 . Like for forward model, the goal here is to learn the mapping φ from o t to s t through two steps, encoding for o t and o t+1 and action prediction. The error is computed for action prediction between a t andâ t and then back-propagated to learn the encodinĝ Learning such model enforces that the state encodes enough information to recover the action that modified the state (Fig. 4 ). • Using prior knowledge to constrain the state space: A last approach is to handle SRL by using specific constraints or prior knowledge about the functioning, dynamics or physics of the world (besides the constraints of forward and inverse models) such as the temporal continuity or the causality principles that generally reflect the interaction of the agent with objects or in the environment BIB003 . Priors are defined as objective or loss functions L, applied on a set of states s 1:n (Fig. 5) , to be minimized (or maximized) under specific condition c. An example of condition can be enforcing locality or time proximity within the set of states. All these approaches are detailed in Section 3.