State Representation Learning for Control: An Overview <s> Auto-encoders (AE) <s> We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind. <s> BIB001 </s> State Representation Learning for Control: An Overview <s> Auto-encoders (AE) <s> Abstract Dimensionality reduction has been a long-standing research topic in academia and industry for two major reasons. First, in al–most every domain, ranging from biology, social science, economics, to military data processing applications, the increasingly large volume of data is challenging the existing computing capability and raising the computing cost. Second, the notion of “intrinsic structure” allows us to remove some redundant dimensions from high-dimensional observations and reduce it into low-dimensional features without signiﬁcant information loss. Autoencoder, as a powerful tool for dimensionality reduction has been intensively applied in image reconstruction, missing data recovery and classiﬁcation. In this paper, we propose a new structure, folded autoencoder based on symmetric structure of conventional autoencoder, for dimensionality reduction. The new structure reduces the number of weights to be tuned and thus reduces the computational cost. Simulation results over MNIST data benchmark validate the effectiveness of this structure. <s> BIB002 </s> State Representation Learning for Control: An Overview <s> Auto-encoders (AE) <s> For the challenging pole balancing task we propose a system which uses raw visual input data for reinforcement learning to evolve a control strategy. Therefore we use a neural network --- a deep autoencoder --- to encode the camera images and thus the system states in a low dimensional feature space. The system is compared to controllers that work directly on the motor sensor data. We show that the performances of both systems are settled in the same order of magnitude. <s> BIB003 </s> State Representation Learning for Control: An Overview <s> Auto-encoders (AE) <s> Auto-encoder-a tricky three-layered neural network, known as auto-association before, constructs the "building block" of deep learning, which has been demonstrated to achieve good performance in various domains. In this paper, we try to investigate the dimensionality reduction ability of auto-encoder, and see if it has some kind of good property that might accumulate when being stacked and thus contribute to the success of deep learning.Based on the above idea, this paper starts from auto-encoder and focuses on its ability to reduce the dimensionality, trying to understand the difference between auto-encoder and state-of-the-art dimensionality reduction methods. Experiments are conducted both on the synthesized data for an intuitive understanding of the method, mainly on two and three-dimensional spaces for better visualization, and on some real datasets, including MNIST and Olivetti face datasets. The results show that auto-encoder can indeed learn something different from other methods. Besides, we preliminarily investigate the influence of the number of hidden layer nodes on the performance of auto-encoder and its possible relation with the intrinsic dimensionality of input data. <s> BIB004 </s> State Representation Learning for Control: An Overview <s> Auto-encoders (AE) <s> Reinforcement learning provides a powerful and flexible framework for automated acquisition of robotic motion skills. However, applying reinforcement learning requires a sufficiently detailed representation of the state, including the configuration of task-relevant objects. We present an approach that automates state-space construction by learning a state representation directly from camera images. Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models. The resulting controller reacts continuously to the learned feature points, allowing the robot to dynamically manipulate objects in the world with closed-loop control. We demonstrate our method with a PR2 robot on tasks that include pushing a free-standing toy block, picking up a bag of rice using a spatula, and hanging a loop of rope on a hook at various positions. In each task, our method automatically learns to track task-relevant objects and manipulate their configuration with the robot's arm. <s> BIB005
Auto-encoders (AE) are a common tool used to learn state representations that are widely used for dimensionality reduction BIB001 BIB002 BIB004 . Their objective is to output a reproduction of the input. Its architecture is composed by an encoder and a decoder. The encoder projects the input to a latent space representation (often in lower dimension than the input), which is re-projected to the output afterwards by the decoder. In our problem setting, o t is the input, s t the latent representation, andô t is the output. The dimensionality of the latent representation can be chosen depending on the dimension of the state representation we want to learn and enforcing it in such case. The AE will then automatically learn a compact representation by minimizing the reconstruction error between input and output. The usual loss function L to measure the reconstruction error is the mean squared error (MSE) between input and output, computed pixel-wise. However, it can be any norm. Auto-encoders are used in different SRL settings BIB005 BIB003 ; PCA can also be considered as a particular case of auto-encoder  .