State Representation Learning for Control: An Overview <s> Learning a forward model <s> The classical approach to using utility functions suffers from the drawback of having to design and tweak the functions on a case by case basis. Inspired by examples from the animal kingdom, social sciences and games we propose empowerment, a rather universal function, defined as the information-theoretic capacity of an agent's actuation channel. The concept applies to any sensorimotor apparatus. Empowerment as a measure reflects the properties of the apparatus as long as they are observable due to the coupling of sensors and actuators via the environment. Using two simple experiments we also demonstrate how empowerment influences sensor-actuator evolution <s> BIB001 </s> State Representation Learning for Control: An Overview <s> Learning a forward model <s> We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems. <s> BIB002 </s> State Representation Learning for Control: An Overview <s> Learning a forward model <s> Kalman Filters are one of the most influential models of time-varying phenomena. They admit an intuitive probabilistic interpretation, have a simple functional form, and enjoy widespread adoption in a variety of disciplines. Motivated by recent variational methods for learning deep generative models, we introduce a unified algorithm to efficiently learn a broad spectrum of Kalman filters. Of particular interest is the use of temporal generative models for counterfactual inference. We investigate the efficacy of such models for counterfactual inference, and to that end we introduce the"Healing MNIST"dataset where long-term structure, noise and actions are applied to sequences of digits. We show the efficacy of our method for modeling this dataset. We further show how our model can be used for counterfactual inference for patients, based on electronic health record data of 8,000 patients over 4.5 years. <s> BIB003 </s> State Representation Learning for Control: An Overview <s> Learning a forward model <s> We summarize the potential impact that the European Union’s new General Data Protection Regulation will have on the routine use of machine learning algorithms. Slated to take effect as law across the EU in 2018, it will restrict automated individual decision-making (that is, algorithms that make decisions based on user-level predictors) which “significantly affect” users. The law will also effectively create a “right to explanation,” whereby a user can ask for an explanation of an algorithmic decision that was made about them. We argue that while this law will pose large challenges for industry, it highlights opportunities for computer scientists to take the lead in designing algorithms and evaluation frameworks which avoid discrimination and enable explanation. <s> BIB004 </s> State Representation Learning for Control: An Overview <s> Learning a forward model <s> For many tasks, tactile or visual feedback is helpful or even crucial. However, designing controllers that take such high-dimensional feedback into account is non-trivial. Therefore, robots should be able to learn tactile skills through trial and error by using reinforcement learning algorithms. The input domain for such tasks, however, might include strongly correlated or non-relevant dimensions, making it hard to specify a suitable metric on such domains. Auto-encoders specialize in finding compact representations, where defining such a metric is likely to be easier. Therefore, we propose a reinforcement learning algorithm that can learn non-linear policies in continuous state spaces, which leverages representations learned using auto-encoders. We first evaluate this method on a simulated toy-task with visual input. Then, we validate our approach on a real-robot tactile stabilization task. <s> BIB005 </s> State Representation Learning for Control: An Overview <s> Learning a forward model <s> We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning of latent Markovian state space models. Leveraging recent advances in Stochastic Gradient Variational Bayes, DVBF can overcome intractable inference distributions by means of variational inference. Thus, it can handle highly nonlinear input data with temporal and spatial dependencies such as image sequences without domain knowledge. Our experiments show that enabling backpropagation through transitions enforces state space assumptions and significantly improves information content of the latent embedding. This also enables realistic long-term prediction. <s> BIB006 </s> State Representation Learning for Control: An Overview <s> Learning a forward model <s> We propose position-velocity encoders (PVEs) which learn---without supervision---to encode images to positions and velocities of task-relevant objects. PVEs encode a single image into a low-dimensional position state and compute the velocity state from finite differences in position. In contrast to autoencoders, position-velocity encoders are not trained by image reconstruction, but by making the position-velocity representation consistent with priors about interacting with the physical world. We applied PVEs to several simulated control tasks from pixels and achieved promising preliminary results. <s> BIB007
Last subsection reviewed how reconstructing an observation is useful to learn state representations. Now we will review how temporal dynamics of the system can also help the same purpose. Therefore we present approaches that rely on learning a forward model to learn a state space. The general idea is to force states to efficiently encode the information necessary to predict the next state (Fig. 3) . As described in Section 2, in the case of the forward models we study here, the model is used as a proxy for learning s t . The model firstly makes a projection from the observation space to the state space to obtain s t and applies a transition to predictŝ t+1 . The error is computed by comparing the estimated next stateŝ t+1 with the value of s t+1 derived from the next observation o t+1 at the next time step. Note that forward models can benefit from the observation reconstruction objective presented in Section 3.1. As an example, the works previously presented in Section 3.1 BIB004 BIB005 BIB002 , Assael et al., 2015 belong to the auto-encoder category of models. However, they all predict future observations to learn representations and therefore, they as well belong to the family of forward models. The method these works use to combine forward models and auto-encoders consists in mapping o t to s t , and then compute the transition, with the help of a t , to obtainŝ t+1 .ŝ t+1 is then remapped onto the pixel space in form of a vectorô t+1 . The error is then computed pixel-wise betweenô t+1 and o t+1 . One common assumption is that the forward model in the learned state space is linear BIB004 BIB005 . The transition is then just a linear combination of s t and a t as in Eq. 6. W, U and V are either fixed or learned parameters BIB005 . In a similar way, the Embed to Control model (E2C) based on variational auto-encoders uses Eq. 6 to compute the mean µ of a distribution and learn supplementary parameters for the variance σ of the distribution BIB002 . Then,ŝ t+1 is computed with Eq. 7: Using distributions to computeŝ t+1 allows to use the KL-divergence to train the forward model. This method is also used in BIB006 and BIB003 . However, the transition model in BIB003 considers the KL-divergence between P (s t+1 ) and P (ŝ t+1 ) and does not use the loss of the reconstruction based on o t+1 and o t+1 . The use of a t is a common feature in most forward models based approaches. In fact, as several future states s t+1 from a given state are possible, s t alone does not contain enough information to predict s t+1 . The only approach that gets rid of the need for actions assumes that the transition from s t−1 to s t allows to deduce the transition from s t to s t+1 and uses several past states to predict s t+1 BIB004 . Actions are therefore implicit in this approach. Another use of a forward model, connected to an intrinsic curiosity model (ICM) which helps agents explore and discover the environment out of curiosity when extrinsic rewards are sparse or not present at all, is proposed in  . In this model, an intrinsic reward signal is computed from the forward model's loss function L f wd (f is the forward function learned by the model,φ is the encoding model): It is argued that there is no incentive in this model for s t to learn to encode any environmental features that cannot influence or are not influenced by the agent's actions. The learned exploration strategy of the agent is therefore robust to uncontrollable aspects of the environment such as the presence of distractor objects, changes in illumination, or other sources of noise in the environment  . Forward models are therefore able to learn representations of controllable factors: in order to predict next state, the model must understand the object being controlled. This kind of representation can also be learned through a controllability prior BIB007 . If a robot acts by applying forces, controllable things should be those whose accelerations correlate with the actions of the robot. Accordingly, a loss function can be defined to minimize the covariance between an action dimension i and the accelerations in the state dimension i. The following formula from BIB007 ] makes it explicit: where cov(a t,i , s t+1,i ) is the covariance between the state s t+1,i at dimension i and time t and a t,i (the action at dimension i that led to such state). Note that here the learned state is assumed to represent an acceleration. Related with this prior also is the notion of empowerment BIB001 , defined as an information-theoretic capacity of an agent's actuation channel to influence its own evolution. The concept of empowerment is related to accountability or agency, i.e., recognizing when an agent is responsible for originating the change of state in the environment.