Model predictive rollouts.
Solving the MPC optimization problem of (REF ) is prohibitively expensive and hard to obtain online. This limitation is overcome by the information theoretic model predictive path integral (i-MPPI) method [1]}, a sampling-based algorithm that uses an approximation of the true dynamics, and is able to optimize both convex and non-convex cost criteria, thus being applicable to large classes of stochastic systems and representations. i-MPPI uses the free energy of the system and relative entropy (KL-divergence) for providing generalized path integral expressions. Crucially, it uses importance sampling for acquiring the optimal control paths. In MoPAC, we use a similar setting for generating model-based rollouts that will profit from the optimal performance guarantees of Sec. . In our model predictive rollouts (MPR), we initialize the control sequence employing the learned policy \(\pi _{\theta }\)  to sample initial actions. Then, we design a similar setup as in i-MPPI for creating trajectory simulations in a specific time horizon \(H\) , starting from an initial true state (Alg. REF ). We evaluate the rewards of the rollouts, using also a VF for the final state (REF ), and we perform importance sampling of the optimal transitions over all simulations, along with an information-theoretic update of the actions' exploration noise. Finally, we collect the resulting optimal transitions in the model-based replay memory \(D_{model}\) .
