The idea of training an RL agent using simulations is present in prior work. The Dyna Architecture uses an action model to train an agent in “hypothetical" states [1]}. Normalized advantage functions (NAF) uses iteratively refitted local linear models for training agents in “imaginations" using Q-learning [2]}. World Models is a model-based policy learning architecture, where (a) a Convolutional Variational Autoencoder (CVAE) is used to learn a latent state representation, (b) a Mixture Density Network output from a Long Short Term Memory (LSTM) network is used to learn a predictive model of the environment, and (c) Evolution Strategies are used to learn a policy in a simple controller using “dreams" [3]}. The NCA is closely related to World Models but differs in that we use a simpler Convolutional Autoencoder (CAE) for state representation learning, the predictive model is learned by a DNC instead of LSTM, the predicted states are output directly instead of being sampled through a Gaussian Mixture Model, and the agent is a feedforward ANN trained using PPO.
