Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems.


Introduction
Control of non-linear dynamical systems with continuous state and action spaces is one of the key problems in robotics and, in a broader context, in reinforcement learning for autonomous agents. A prominent class of algorithms that aim to solve this problem are model-based locally optimal (stochastic) control algorithms such as iLQG [1,2], which approximate the general non-linear control problem via local linearization. When combined with receding horizon control [3], and machine learning methods for learning approximate system models, such algorithms are a powerful tool for solving complicated control problems [3,4,5,6]; however, they either rely on a known system model or require the design of relatively low-dimensional state representations. For real autonomous agents to succeed, we ultimately need algorithms that are capable of controlling complex dynamical systems from raw sensory input (e.g. images) only. In this paper we tackle this difficult problem.
If stochastic optimal control (SOC) methods were applied directly to control from raw image data they would face two major obstacles. First, sensory data is usually high-dimensional -i.e. images with thousands of pixels -rendering a naive SOC solution computationally infeasible. Second, the image content is typically a highly non-linear function of the system dynamics underlying the observations; thus model identification and control of this dynamics is non-trivial.
While both problems could, in principle, be addressed by designing more advanced SOC algorithms we approach the "optimal control from raw images" problem radically different: turning the problem of locally optimal control in high-dimensional, non-linear systems, into one of identifying a lowdimensional latent state space in which locally optimal control can be performed robustly and easily. To learn such a latent space we propose a new deep generative model belonging to the class of variational autoencoders [7,8] that is derived directly from an iLQG formulation in latent space. The resulting Embed to Control (E2C) system is a probabilistic generative model that holds a belief over viable trajectories in sensory space; allows for accurate long-term planning in latent space; and is trained fully unsupervised.
We demonstrate the success of our approach on four challenging tasks for control from raw images and compare it to a range of methods for unsupervised representation learning. As an aside, we also validate that deep up-or de-convolutional networks [9,10,11] can be used as powerful generative models for large images.

The Embed to Control (E2C) model
We briefly review the problem of SOC for dynamical systems, introduce approximate locally optimal control in latent space, and finish with the derivation of our model.

Problem Formulation
We consider the control of unknown dynamical systems of the form where t denotes the time steps, s t ∈ R ns the system state, u t ∈ R nu the applied control and ξ the system noise. The function f (s t , u t ) is an arbitrary, smooth, system dynamics. We equivalently refer to Equation (1) using the notation P (s t+1 |s t , u t ), which we assume to be a multivariate normal distribution N (f (s t , u t ), Σ ξ ). We further assume that we are only given access to visual depictions x t ∈ R nx of state s t . This restriction requires solving a joint state identification and control problem.
For simplicity we will in the following assume that x t is a fully observed depiction of s t , but relax this assumption later.
Our goal then is to infer a low-dimensional latent state space model in which optimal control can be performed. That is, we seek to learn a function m, mapping from high-dimensional images x t to low-dimensional vectors z t ∈ R nz with n z n x , such that the control problem can be solved in z t instead of x t : where ω accounts for system noise; or equivalently z t ∼ N (Z t |m(x t ), Σ ω ). Assuming for the moment that such a function can be learned (or approximated), we will first define SOC in a latent space and introduce our model thereafter.

Stochastic locally optimal control in latent spaces
Let z t ∈ R nz be the inferred latent state from image x t of state s t and f lat (z t , u t ) the transition dynamics in latent space, i.e., z t+1 = f lat (z t , u t ). Thus f lat models the changes that occur in z t when control u t is applied to the underlying system as a latent space analogue to f (s t , u t ).
Assuming f lat is known, optimal controls for a trajectory of length T in the dynamical system can be derived by minimizing the function J(z 1:T , u 1:T ) which gives the expected future costs when following (z 1:T , u 1:T ): where c(z t , u t ) are instantaneous costs, c T (z T , u T ) denotes terminal costs and z 1:T = {z 1 , . . . , z T } and u 1:T = {u 1 , . . . , u T } are state and action sequences respectively. If z t contains sufficient information about s t , i.e., s t can be inferred from z t alone, and f lat is differentiable, the cost-minimizing controls can be computed from J(z 1:T , u 1:T ) via SOC algorithms [12]. These optimal control algorithms approximate the global non-linear dynamics with locally linear dynamics at each time step t. Locally optimal actions can then be found in closed form. Formally, given a reference trajectorȳ z 1:T -the current estimate for the optimal trajectory -together with corresponding controlsū 1:T the system is linearized as where A(z t ) = δf lat (zt,ūt) δzt , B(z t ) = δf lat (zt,ūt) δūt are local Jacobians, and o(z t ) is an offset. To enable efficient computation of the local controls we assume the costs to be a quadratic function of the latent representation Figure 1: The information flow in the E2C model. From left to right, we encode and decode an image x t with the networks h enc φ and h dec θ , where we use the latent code z t for the transition step. The h trans ψ network computes the local matrices A t , B t , o t with which we can predictẑ t+1 from z t and u t . Similarity to the encoding z t+1 is enforced by a KL divergence on their distributions and reconstruction is again performed by h dec θ .
where R z ∈ R nz×nz and R u ∈ R nu×nu are cost weighting matrices and z goal is the inferred representation of the goal state. We also assume c T (z t , u t ) = c(z t , u t ) throughout this paper.
In combination with Equation (4) this gives us a local linear quadratic Gaussian formulation at each time step t which can be solved by SOC algorithms such as iterative linear quadratic regulation (iLQR) [13] or approximate inference control (AICO) [14]. The result of this trajectory optimization step then is a locally optimal trajectory with corresponding control sequence (z * 1:T , u * 1:T ) ≈ arg minz1:T u 1:T J(z 1:T , u 1:T ).

A locally linear latent state space model for dynamical systems
Starting from the SOC formulation, we now turn to the problem of learning an appropriate lowdimensional latent representation z t ∼ P (Z t |m(x t ), Σ ω ) of x t . The representation z t has to fulfill three properties: (1) it must capture sufficient information about x t (enough to enable reconstruction); (2) it must allow for accurate prediction of the next latent state z t+1 and thus, implicitly, of the next observation x t+1 ; (3) the prediction f lat of the next latent state must be locally linearizable for all valid control magnitudes u t . Given some representation z t , properties (2) and (3) in particular require us to capture possibly highly non-linear changes of the latent representation due to transformations of the observed scene induced by control commands. Crucially, these are particularly hard to model and subsequently linearize. We circumvent this problem by taking a more direct approach: instead of learning a latent space z and transition model f lat which are then linearized and combined with SOC algorithms, we directly impose desired transformation properties on the representation z t during learning. We will select these properties such that prediction in the latent space as well as locally linear inference of the next observation according to Equation (4) are easy.
The transformation properties that we desire from a latent representation can be formalized directly from the iLQG formulation given in Section 2.2 . Formally, following Equation (2), let the latent representation be Gaussian P (Z|X) = N (Z|m(x t ), Σ ω ). To infer z t from x t we first require a method for sampling latent states. Ideally, we would generate samples directly from the unknown true posterior P (Z|X), which we, however, have no access to. Following the variational Bayes approach (see Jordan et al. [15] for an overview) we resort to sampling z t from an approximate posterior distribution Q φ (Z|X) with parameters φ.
Inference model for Q φ . In our work this is always a diagonal Gaussian distribution Q φ (Z|X) = N (Z|µ t , diag(σ 2 t )), whose mean µ t ∈ R nz and covariance Σ t = diag(σ 2 t ) ∈ R nz×nz are computed by an encoding neural network with outputs where h enc φ ∈ R ni is the activation of the last hidden layer and where φ is given by the set of all learnable parameters of the encoding network, including the weight matrices W µ , W σ and biases b µ , b σ . Parameterizing the mean and variance of a Gaussian distribution based on a neural network gives us a natural and very expressive model for our latent space. It additionally comes with the benefit that we can use the reparameterization trick [7,8] to backpropagate gradients of a loss function based on samples through the latent distribution.
Generative model for P θ . Using the approximate posterior distribution Q φ we generate observed samples (images)x t andx t+1 from latent samples z t and z t+1 by enforcing a locally linear relationship in latent space according to Equation (4), yielding the following generative model: whereQ ψ is the next latent state posterior distribution which exactly follows the linear form required for stochastic optimal control, ω t ∼ N (0, H t ) is an estimate of the system noise and C can be decomposed as C t = A t Σ t A T t + H t . Note that while the transition dynamics in our generative model operates on the inferred latent space, it takes untransformed controls into account. That is, we aim to learn a latent space such that the transition dynamics in z linearizes the non-linear observed dynamics in x and is locally linear in the applied controls u. Reconstruction of an image from z t is performed by passing the sample through multiple hidden layers of a decoding neural network which computes the mean p t of the generative Bernoulli distribution 1 where h dec θ (z t ) ∈ R n d is the response of the last hidden layer in the decoding network. The set of parameters for the decoding network, including weight matrix W p and b p , then make up the learned generative parameters θ.
Transition model forQ ψ . What remains is to specify how the linearization matrices A t ∈ R nz×nz , B t ∈ R nz×nu and offset o t ∈ R nz are predicted. Following the same approach as for distribution means and covariance matrices, we predict all local transformation parameters from samples z t based on the hidden representation h trans ψ (z t ) ∈ R nt of a third neural network with parameters ψto which we refer as the transformation network. Specifically, we parametrize the transformation matrices and offset as vec where vec denotes vectorization and therefore vec[A t ] ∈ R (n 2 z ) and vec[B t ] ∈ R (nz·nu) . To circumvent estimating the full matrix A t of size n z × n z , we can choose it to be a pertubation of the identity matrix A t = (I − vr T ) −1 which reduces the parameters to be estimated for A t to 2n z .
A sketch of the complete architecture is shown in Figure 1 . It also visualizes the additional constraint that is essential for learning a representation for long-term predictions: we require sampleŝ z t+1 from the state transition distributionQ ψ to be similar to the encoding of x t+1 through Q φ . While it might seem that just learning a perfect reconstruction of x t+1 fromẑ t+1 is enough, we require multi-step predictions for planning in Z which must correspond to valid trajectories in the observed space X. Without enforcing similarity between samples fromQ ψ and Q φ following a transition in latent space from z t using control u t may lead to a pointẑ t+1 from which reconstruction of x t+1 is possible but that is not a valid encoding (i.e. the model has never seen an image corresponding toẑ t+1 ). Executing another action inẑ t+1 then does not result in a valid latent state -since the transition model is conditional on samples coming from the inference network -and thus long-term predictions fail. In a nutshell such a divergence between encodings and the transition model results in a generative model that does not accurately model the Markov chain formed by the observations.

Learning via stochastic gradient variational Bayes
For training the model we use a data set D = {(x 1 , u 1 , x 2 ), . . . , (x T −1 , u T −1 , x T )} containing observation tuples with corresponding controls obtained from interactions with the dynamical system. Using this data set, we learn the parameters of the inference, transition and generative model by minimizing a variational bound on the true data negative log-likelihood − log P (x t , u t , x t+1 ) plus an additional constraint on the latent representation. The complete loss function is given as 1 A Bernoulli distribution for P θ is a common choice when modeling black-and-white images.
The first part of this loss is the per-example variational bound on the log-likelihood (12) where Q φ , P θ andQ ψ are the parametric inference, generative and transition distributions from Section 2.3 and P (Z t ) is a prior on the approximate posterior Q φ ; which we always chose to be an isotropic Gaussian distribution with mean zero and unit variance. The second KL divergence in Equation (11) is an additional contraction term with weight λ, that enforces agreement between the transition and inference models. This term is essential for establishing a Markov chain in latent space that corresponds to the real system dynamics (see Section 2.3 above for an in depth discussion). This KL divergence can also be seen as a prior on the latent transition model. Note that all KL terms can be computed analytically for our model 2 .
During training we approximate the expectation in L(D) via sampling. Specifically, we take one sample z t for each input x t and transform that sample using Equation (10) to give a valid sampleẑ t+1 fromQ ψ . We then jointly learn all parameters of our model by minimizing L(D) using stochastic gradient descent. We highlight that training proceeds without any layer-wise pre-training, alternating optimization or other tricks.

Experimental Results
We evaluate our model on four visual tasks: an agent in a plane with obstacles, a visual version of the classic inverted pendulum swing-up task, balancing a cart-pole system, and control of a 3-link arm with larger images. These are described in detail below.

Experimental Setup
Model training. We consider two different network types for our model: Standard fully connected neural networks with up to three layers which work well for moderately sized images and are used for the planar and swing-up experiments; A deep convolutional network for the encoder in combination with an "up-convolutional" (or deconvolutional) network as the decoder which, in accordance to recent findings from the literature [9,10,11], we found to be an adequate model for larger images. Training was performed using Adam [16] throughout all experiments. The training data set D for all tasks was generated by randomly sampling N state observations and actions with corresponding successor states. For the plane we used N = 3, 000 samples, for the inverted pendulum and cart-pole system we used N = 15, 000 and for the 3-link arm N = 30, 000. A complete list of architecture parameters and hyperparameter choices as well as an in-depth explanation of the up-convolutional network are specified in the supplementary material. To facilitate reproduction of the results presented we will open source our implementation in Theano [17] together with the considered benchmarks.
Model variants. In addition to the Embed to Control (E2C) dynamics model derived above we also consider two variants: By removing the latent dynamics network h trans ψ i.e. setting its output to one in Equation (10) -we obtain a variant in which A t , B t and o t are estimated as globally linear matrices (Global E2C). If we instead replace the transition model with a network estimating the dynamics as a non-linear functionf lat and only linearize during planning, estimating A t , B t , o t as Jacobians tô f lat as described in Section 2.2, we obtain a variant with nonlinear latent dynamics.
Baseline models. For a thorough comparison, and to exhibit the complicated nature of the tasks, we also test a set of baseline models on the plane and the inverted pendulum task: a standard variational autoencoder (VAE) and a deep autoencoder (AE) are trained on the autoencoding subtask for visual problems. That is, given a data set D used for training our model, we remove all actions from the tuples in D and disregard temporal context between images. After autoencoder training we learn a dynamics model in latent space, approximating f lat from Section 2.2. We also consider a VAE variant with a slowness term on the latent representation. Due to space constraints we refer the reader to the supplementary material for a detailed derivation of this model. The architectures used for the baseline methods are the same as for our E2C model in all experiments.  Optimal control algorithms. To perform optimal control in the latent space of different models, we employ two trajectory optimization algorithms: iterative linear quadratic regulation (iLQR) [13] (for the plane and inverted pendulum) and approximate inference control (AICO) [14] (all other experiments). For all VAEs both methods operate on the mean of distributions Q φ andQ ψ . AICO additionally makes use of the local Gaussian covariances Σ t and C. Except for the experiments on the planar system, control was performed in a model predictive control fashion using the receding horizon scheme introduced in [3]. To obtain closed loop control given an image x t , it is first passed through the encoder to obtain the latent state z t . A locally optimal trajectory is subsequently found by optimizing (z * t:t+T , u * t:t+T ) ≈ arg minzt:t+T u t:t+T J(z t:t+T , u t:t+T ) with fixed, small horizon T (with T = 10 unless noted otherwise). Controls u * t are applied to the system and a transition to z t+1 is observed (by encoding the next state image x t+1 ). Then a new control sequence -with horizon T -starting in z t+1 is found using the last estimated trajectory as a bootstrap. Note that planning is performed entirely in the latent state without access to any observations except for the depiction of the current state. To compute the cost function c(z t , u t ) required for trajectory optimization in z we assume knowledge of the observation x goal of the goal state s goal . This observation is then transformed into latent space and costs are computed according to Equation (5).

Control in a planar system
The agent in the planar system can move in a bounded two-dimensional plane by choosing a continuous offset in x-and y-direction. Obstructed by six circular obstacles, the task is to move to the bottom right of the image, starting from a random x position at the top of the image. The highdimensional representation of a state is a 40 × 40 black-and-white image. The agent cannot move beyond boundaries. The cost matrices were set to be diagonal with diag(R z ) = 0.1, diag(R u ) = 1. The encodings of obstacles are obtained prior to planning and an additional quadratic cost term penalizing closeness to obstacles was added to the cost based on them.
A depiction of the observations on which control is performed -together with their corresponding state values and embeddings into latent space -is shown in Figure 2. The figure also clearly shows a fundamental advantage the E2C model has over its competitors: While the separately trained autoencoders make for aesthetically pleasing pictures, the models failed to discover the underlying structure of the state space, complicating dynamics estimation and largely invalidating costs based on distances in said space. Including the latent dynamics constraints in these end-to-end models on the other hand, yields latent spaces approaching the optimal planar embedding.
We test the long-term accuracy by accumulating latent and real trajectory costs to quantify whether the imagined trajectory reflects reality. The results for all models when starting from random positions and executing 50 pre-computed actions are summarized in Table 1 -using a seperate test set for evaluating reconstructions. While all methods achieve a low mean reconstruction log loss per image, the difference in accumulated costs per trajectory clearly show the superiority of the E2C model. Using the globally or locally linear E2C model, trajectories planned in latent space are as Table 1: Comparison between different approaches to model learning from raw pixels for the planar and inverted pendulum system. We compare all models with respect to their prediction quality on a test set of sampled transitions and with respect to their performance when combined with SOC (trajectory cost for control from different start states). Note that trajectory costs in latent space are not necessarily comparable. The "real" trajectory cost was computed on the dynamics of the simulator while executing planned actions. For the true models for s t , real trajectory costs were 20.24 ± 4.15 for the planar system, and 9.8 ± 2.4 for the inverted pendulum. A † marks separately trained dynamics networks. good as trajectories planned on the real system state. All models besides E2C fail to give long-term predictions that result in good performance.

Learning swing-up for an inverted pendulum
We next turn to the task of controlling the classical inverted pendulum system [18,19] from images. We create depictions of the state by rendering a fixed length line starting from the center of the image at an angle corresponding to the pendulum position. The goal in this task then is to swing-up and balance an underactuated pendulum from a resting position, in which the pendulum is hanging down. Exemplary observations and reconstructions for this system are given in Figure 3 (d). In the visual inverted pendulum task our algorithm faces two additional difficulties: the observed space is non-Markov, as the angular velocity cannot be inferred from a single image, and second, discretization errors due to rendering pendulum angles as small 48x48 pixel images make exact control difficult.
To restore the Markov property, we stack two images (increasing the number of input channels), thus observing a one-step history.  can learn a meaningful embedding, cleanly seperating velocities from positions, from this noisy data is remarkable; and no other model recovered this shape. Table 1 again compares the different models quantitatively. While the E2C model is not the best in terms of reconstruction performance, it is the only model resulting in stable swing-up and balance behavior. The only other model that can be used to swing-up an inverted pendulum is the E2C model with non-linear dynamics, which however does not result in a control that can balance the pole once swung-up. We explain this failure with the fact that the non-linear latent dynamics model cannot be guaranteed to be linearizable for all control magnitudes, resulting in undesired behavior around unstable fixpoints of the real system dynamics. Also note that for this task a globally linear dynamics model is inadequate.

Balancing a cart-pole and controlling a simulated robot arm
Finally, we consider control of two more complex dynamical systems from high-dimensional images using a six layer convolutional inference and six layer up-convolutional generative network, resulting in a 12-layer deep path from input to reconstruction. Specifically, we control a visual version of the classical cart-pole system [20] from a history of two raw 80 × 80 pixel images as well as a three-link planar robot arm based on a history of two 128 × 128 pixel images. The latent space was set to be 8 dimensional in both experiments. The real state dimensionality for the cart-pole is four and is controlled using one action while for the arm the real state can be described in 6 dimensions (joint angles and velocities) and controlled using a three-dimensional action vector corresponding to motor torques. We simply used cost weights diag(R z ) = 1, diag(R u ) = 1 in both experiments.
As in previous experiments the E2C model seems to have no problem finding a locally linear embedding of images into latent space in which control can be performed. Figure 4 depicts exemplary images -for both problems -from a trajectory executed by our system when combined with MPC. The costs for these trajectories (11.13 for the cart-pole, 85.12 for the arm) are only slightly worse than trajectories obtained by AICO operating on the real system dynamics starting from the same start-state (7.28 and 60.74 respectively). The supplementary material contains additional experiments using these domains. A video containing exemplary controlled trajectories for all systems can be found at http://ml.informatik.uni-freiburg.de/research/e2c .

Comparison to recent work
In the context of representation learning for control (see Böhmer et al. [21] for a recent review), deep autoencoders (ignoring state transitions) similar to those used in our comparison have been applied previously, e.g. by Lange and Riedmiller [22]. A more direct route to achieve control based on image streams is taken by recent work employing end-to-end reinforcement learning from raw images. This approach has been used by Mnih et al. [23], who learn deep non-linear Q-functions and Levine et al. [24] who learn a deep policy mapping directly to motor torques.
Close to the approach we propose is a recent paper by Wahlström et al. [25], where autoencoders are used to extract a latent representation from images, on which a non-linear model of the (latent) forward dynamics is learned with MLPs. This model is then used in combination with MPC. The autoencoder and dynamics model are trained jointly and are thus similar to the non-linear E2C variant in our comparison. In contrast to our model the approach however requires pre-processing by PCA and does neither ensure that long-term predictions in latent space do not diverge, nor that they are linearizable.
As stated above, our system belongs to the family of VAEs and is generally similar to recent work such as Kingma and Welling [7], Rezende et al. [8], Gregor et al. [26], Bayer and Osendorfer [27]. While we learn a latent dynamics model for control, our approach shares with Gregor et al. [28] another recent stochastic VAE variant -the use of a history of image sequences to learn the latent representation. Two additional parallels between our work and recent advances for training deep neural networks can be observed. First, the idea of enforcing desired transformations in latent space during learning, such that the the data is easy to model in that space, has appeared several times already in the literature. This includes the development of transforming auto-encoders [29] and recent probabilistic models for images [30,31]. Second, learning relations between pairs of images -although without control -has received considerable attention from the community during the last years [32,33]. We refer to Memisevic [33] for a recent overview and note that the predicted transition matrices A t and B t in the E2C algorithm can implement the multiplicative interactions typically found in these models.
In a broader context our model is also related to work on learning non-linear (latent) dynamic models for observations in a Markov decision process through, e.g., hidden Markov models and Kalman filters [34,35,36]. We refer to Langford et al. [37] for a recent discussion and a general formalization of this problem.

Conclusion
We presented Embed to Control (E2C), a system for stochastic optimal control on high-dimensional image streams. Key to the approach is the extraction of a latent dynamics model which is constrained to be locally linear in its state transitions. An evaluation on four challenging benchmarks revealed that E2C can find embeddings on which control can be performed with ease, reaching performance close to that achievable by optimal control on the real system model.
is simply not an option. We thus decided to use deconvolution or up-convolutional networks which were recently show to be powerful models for image generation [9,10,11].
To set-up these models we basically "mirror" the convolutional architecture used for the encoder. More specifically for each 5 × 5 convolution followed by 2 × 2 max-pooling step in the encoder network we introduce a 5 × 5 convolution and 2 × 2 up-sampling step in the decoder network. The complete network architecture is given below. It is similar to the up-convolution networks used in Dosovitskiy et al. [10]. As the up-sampling strategy we used simple "perforated" up-sampling as described in [38].

B.2 Variational Autoencoder with slowness
Enforcing temporal slowness during learning has previously been found to be a good proxy for learning representations in reinforcement learning [39,40], and representation learning from videos [41]. We also consider a VAE variant with a slowness term on the latent representation by enforcing similarity of the encodings of temporally close images. This can be achieved by augmenting the standard VAE objective L bound with an additional KL divergence term on the latent posterior Q φ : Indeed there seems to be a slightly better coherence of similar states in the latent spaces, as e.g. depicted in Figure 8 in the main paper. Yet, our experiments show that a slowness term alone does not suffice to structure the latent space such that locally linear predictions and control become feasible.

B.3 Evaluation criteria
For comparing the performance of all variants of E2C and the baselines, the following criteria are of importance: • Autoencoding. Being able to reconstruct the given observations is the basic necessity for a model to work. The reconstruction cost drives a model to identify single states from its observations. • Decoding the next state. For any planning to be possible at all, the decoder must be able to generate the correct images from the transitions the dynamics model performed. If this is not the case, we know that the latent states of the encoding and the transition model do not coincide, thus preventing any planning. • Optimizing latent trajectory costs. The action sequences for achieving a specified goal will be determined completely by locally linearized dynamics in the latent space. Therefore minimizing trajectory costs in latent space is, again, a necessity for successful control. • Optimizing real trajectory costs. While the action sequence has been determined for the latent dynamics, the deciding criterion is whether this reflects the true state trajectory costs. Therefore carrying out the "dreamed" plans in reality is the optimality criterion for every model.
We reflected these four criteria in the evaluation table in the paper. For the reconstruction of the current and next state we specified the mean log loss, which is in case of the Bernoulli distributions the cross entropy error function: For the costs a model imagines and truly achieves, we sample from different starting states and accumulate the distances in latent and true state space according to the SOC method.

B.4 The three-link robot arm
The robot arm we used in the last experiment in the main paper was simulated using dynamics generated by the MapleSim http://www.maplesoft.com/products/maplesim/ simulator wrapped in Python and visualized for producing inputs to E2C using PyGame. We simulated a fairly standard robot arm with three links. The length of the links were set to 2, 1.2 and 0.7 (units were set to meters). The masses of the corresponding links were all set to 10kg.

B.5 Evaluating the true system model
To compare the efficacy of different models when combined with optimal control algorithms we always reported the cost in latent space (as used by the optimal control algorithm) as well as the "real" trajectory cost. To compute this real cost we computed the same cost function as in the latent space (quadratic costs on the deviation from a given goal state) but using the real system states during execution of the trajectory that was planned entirely in the latent state.
As an upper bound on the performance achievable for control by any of the models we also computed the true system cost by applying iLQR/AICO to a model of the real system dynamics. We have this model available since all experiments were performed in simulation. In order to learn dynamics predictions for the image-only autoencoders, we extracted the latent representations and combined them with the actions from D into On these low-dimensional representations we trained the dynamics MLPs, thus ensuring that all methods were trained on exactly the same data.

B.6.2 Implementation details
We used orthogonal weight initialization for every layer [42]. As described in the main paper Adam [16] was used as the learning rule for all networks. We found both these techniques to be fundamentally important for stabilizing training and achieving good reconstructions for all methods. Both methods also clearly helped to cut the hyperparameter search needed for all methods to a minimum. In the process of training, we could make out three phases: the unfolding of the latent space, the overcoming of the trivial solution (the average image of the dataset) and the minimization of the latent KL term.  Figure 5: Generated "dreamed" trajectories of different models for the plane task (from left to right). The opacity of the obstacles has been lowered in this depiction for better visibility of the agent.

C Supplementary evaluations C.1 Trajectories for plane and pendulum
To qualitatively measure the predictive accuracy, the starting state for a trajectory is encoded and the actions are applied on the latent representation. After each transition, the predicted latent position is decoded and visualized. In this manner, multi-step predictions can be generated for the planar system in Figure 5 and for the inverted pendulum in Figures 6 and 7.  Figure 6: Generated "dreamed" trajectories (from left to right) for passive dynamics: the pendulum starts with angle θ = − π 2 without velocity. The models have to predict the dynamics, while no force is applied.  Figure 7: Dreamed trajectories (from left to right) for controlled dynamics: the pendulum starts with angle θ = π 2 without velocity. For 6 timesteps, full force is applied to the right, followed by 4 timesteps of full force to the left.

C.3 Trajectories for cart-pole and three-link arm
Finally -similar to the images in Section C.1 - Figure 9 shows multi-step predictions for the cartpole system. We depict show important cases, a long-term prediction with the cart-pole standing still (essentially the unstable fix-point of the underlying dynamics). The cart-pole moving to the right, changing the direction of the poles angular velocity (middle column) and the pole moving farthest to the right. The long-term predictions by the E2C model are all of high quality. Note that for the uncontrolled dynamics the predictions show a slight bias of the pole moving to the right (an effect that we consistently saw in trained models for the cart-pole). We attribute this problem to the fact that discretization errors when rendering the pole angle make it hard to predict small velocities accurately.

Moving left Moving right
Real Generated E2C Real Generated E2C Real Generated E2C Figure 9: Dreamed trajectories (top to bottom) for uncontrolled (left column) and controlled (middle/right column) dynamics in the cart-pole system. The red image shows the initial configuration which is encoded to give z 1 the images in the right part of each column are then generated without additional input by following the dynamics in latent space. The left column depicts the uncontrolled case (u = 0 for all steps). The middle column shows a controlled trajectory with torque −20 applied in each step and the right column a trajectory with torque 20 applied in each step. Prediction of the history image is omitted in these images.
C.4 Exemplary trajectory taken for three-link arm task Figure 10 shows a segment of a controlled trajectory as executed by the E2C system. Note that, in contrast to other figures in this supplementary material, it does not show a long-term prediction but rather 10 steps of a trajectory (together with one-step-ahead predictions) that was taken by the E2C system when combined with model predictive control. For additional visualizations and controlled trajectories for all tasks we refer to the supplementary video.

C.5 Comparison of trajectory optimizers for cart-pole and robot arm
To compare how well AICO deals with the covariance matrices estimated in latent space we performed an additional experiment using on the cart-pole and three-link robot arm task comparing it to iLQR. We performed model predictive control using the locally linear E2C model starting in 10 different start states each. For the cart-pole this was always the goal state (zero angle and zero velocity) with small gaussian noise with variance 0.01. For the three-link arm system this was always a random starting position in the configuration space (while the goal always was to unroll all robot joints (e.g. make all angles zero).
As reported in Table C.5 both methods performed about the same for these tasks with a slight advantage for AICO.