State Representation Learning for Control: An Overview <s> Variational auto-encoders (VAE) <s> We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems. <s> BIB001 </s> State Representation Learning for Control: An Overview <s> Variational auto-encoders (VAE) <s> Kalman Filters are one of the most influential models of time-varying phenomena. They admit an intuitive probabilistic interpretation, have a simple functional form, and enjoy widespread adoption in a variety of disciplines. Motivated by recent variational methods for learning deep generative models, we introduce a unified algorithm to efficiently learn a broad spectrum of Kalman filters. Of particular interest is the use of temporal generative models for counterfactual inference. We investigate the efficacy of such models for counterfactual inference, and to that end we introduce the"Healing MNIST"dataset where long-term structure, noise and actions are applied to sequences of digits. We show the efficacy of our method for modeling this dataset. We further show how our model can be used for counterfactual inference for patients, based on electronic health record data of 8,000 patients over 4.5 years. <s> BIB002 </s> State Representation Learning for Control: An Overview <s> Variational auto-encoders (VAE) <s> For many tasks, tactile or visual feedback is helpful or even crucial. However, designing controllers that take such high-dimensional feedback into account is non-trivial. Therefore, robots should be able to learn tactile skills through trial and error by using reinforcement learning algorithms. The input domain for such tasks, however, might include strongly correlated or non-relevant dimensions, making it hard to specify a suitable metric on such domains. Auto-encoders specialize in finding compact representations, where defining such a metric is likely to be easier. Therefore, we propose a reinforcement learning algorithm that can learn non-linear policies in continuous state spaces, which leverages representations learned using auto-encoders. We first evaluate this method on a simulated toy-task with visual input. Then, we validate our approach on a real-robot tactile stabilization task. <s> BIB003 </s> State Representation Learning for Control: An Overview <s> Variational auto-encoders (VAE) <s> Learning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificial intelligence that is able to learn and reason in the same way that humans do. We introduce beta-VAE, a new state-of-the-art framework for automated discovery of interpretable factorised latent representations from raw image data in a completely unsupervised manner. Our approach is a modification of the variational autoencoder (VAE) framework. We introduce an adjustable hyperparameter beta that balances latent channel capacity and independence constraints with reconstruction accuracy. We demonstrate that beta-VAE with appropriately tuned beta > 1 qualitatively outperforms VAE (beta = 1), as well as state of the art unsupervised (InfoGAN) and semi-supervised (DC-IGN) approaches to disentangled factor learning on a variety of datasets (celebA, faces and chairs). Furthermore, we devise a protocol to quantitatively compare the degree of disentanglement learnt by different models, and show that our approach also significantly outperforms all baselines quantitatively. Unlike InfoGAN, beta-VAE is stable to train, makes few assumptions about the data and relies on tuning a single hyperparameter, which can be directly optimised through a hyper parameter search using weakly labelled data or through heuristic visual inspection for purely unsupervised data. <s> BIB004
The SRL literature has also benefited from the variational inference used in variational autoencoders (VAE)  Welling, 2013, Jimenez Rezende et al., 2014] to learn a mapping from observations to state representations. A VAE is an auto-encoder with probabilistic hidden cells: it interprets S as a set sampled from distribution P (s t |o t ). It then approximates P (s t |o t ) with a model q θ (s t |o t ) called the approximate posterior or recognition model. θ represents the parameters of the model, which, for instance, can be a neural network. The VAE also provides a generator which approximates P (o t |s t ) with a model p φ . φ represents the parameters of the generator. Both models p and q are then trained by minimizing the error between o t andô t and the KL divergence between q θ (s t |o t ) and the normal distribution N (µ = 0, σ = I) (where µ is the mean of the distribution, σ its covariance matrix and I the identity matrix). VAE-related models that do not use exactly the original VAE, but variations of it, are BIB001 , Assael et al., 2015 BIB002 BIB003 BIB004 .