Deep Surrogate Assisted Generation of Environments
Recent progress in reinforcement learning (RL) has started producing
generally capable agents that can solve a distribution of complex environments.
These agents are typically tested on fixed, human-authored environments. On the
other hand, quality diversity (QD) optimization has been proven to be an
effective component of environment generation algorithms, which can generate
collections of high-quality environments that are diverse in the resulting
agent behaviors. However, these algorithms require potentially expensive
simulations of agents on newly generated environments. We propose Deep
Surrogate Assisted Generation of Environments (DSAGE), a sample-efficient QD
environment generation algorithm that maintains a deep surrogate model for
predicting agent behaviors in new environments. Results in two benchmark
domains show that DSAGE significantly outperforms existing QD environment
generation algorithms in discovering collections of environments that elicit
diverse behaviors of a state-of-the-art RL agent and a planning agent.