In this work, we introduce Model Predictive Actor-Critic (MoPAC)–an algorithm that seamlessly combines the sample efficiency of MBRL, with MF actor-critic methods for improved exploration. To ensure exploitation of the learned policy, we propose the use of model predictive rollouts, a method that inherits properties of model predictive path integrals [1]}, capitalizing on the free-energy of the learned system and leveraging the information theoretic constraint in the MB simulations. Our novel method is theoretically sound, as we provide a bound on the performance of trajectory optimization through MPC, when approximating the dynamics model and the value-function, that potentially allows planning for longer time-horizons. Moreover, the maximum entropy objective [2]} in the policy optimization counterbalances the exploitation of model predictive rollouts with exploration on the real environment, enabling the learning of more expressive model dynamics, together with approximating the optimal policy. Our empirical results, both in simulated control tasks and on a physical robotic hand that performs in-hand manipulation, showcase the accelerated learning that MoPAC offers against representative baselines, evincing its effectiveness for learning complex skills on real-robot platforms.
