We present an artificial intelligence research platform inspired by the human game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, a.k.a. MMOs). We demonstrate how this platform can be used to study behavior and learning in large populations of neural agents. Unlike currently popular game environments, our platform supports persistent environments, with variable number of agents, and open-ended task descriptions. The emergence of complex life on Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for finite resources. Our platform aims to simulate this setting in microcosm: we conduct a series of experiments to test how large-scale multiagent competition can incentivize the development of skillful behavior. We find that population size magnifies the complexity of the behaviors that emerge and results in agents that out-compete agents trained in smaller populations. Life on Earth can be viewed as a massive multiagent competition. The cheetah evolves an aerodynamic profile in order to catch the gazelle, the gazelle develops springy legs to run even faster: species have evolved ever new capabilities in order to outcompete their adversaries.The success of biological evolution has inspired many attempts to emulate it in silico, ranging from genetic algorithms that bear only loose resemblance to natural processes, to full-blown simulations of "artificial life". A recurring question has been: at what level of abstraction should we simulate the competitive game of life?In recent years, the field of deep reinforcement learning (RL) has embraced a related approach: train algorithms by having them compete in simulated games BID16 BID14 BID8 . Such games are immediately interpretable and provide easy metrics derived from the game's "score" and win conditions. However , popular game benchmarks are currently still limited: they typically define a narrow, episodic task, with a small fixed number of players. In contrast , life on Earth involves a persistent environment, an unbounded number of players, and a seeming "open-endedness", where ever new and more complex species emerge over time, with no end in sight BID18 .Our aim is to develop a simulation platform (see FIG3 ) that captures important properties of life on Earth, while also borrowing from the interpretability and abstractions of human-designed games. To this end , we turn to the game genre of Massively Multiplayer Online Role-Playing Games (MMORPGs, or MMOs for short). These games involve a large, variable number of players competing to survive and prosper in persistent and far-flung environments. Our platform simulates a "Neural MMO" -an MMO in which each agent is a neural net that learns to survive using RL.We demonstrate the capabilities of this platform through a series of experiments that investigate emergent complexity as a function of the number of agents and species that compete in the simulation. We find that large populations act as competitive pressure that encourages exploration of the environment and the development of skillful behavior. In addition, we find that when agents are organized into species (share policy parameters), each species naturally diverges from the others to occupy its own behavioral niche. Upon publication , we will opensource the platform in full. We alternate between collecting experience across 100 procedurally generated worlds and updating agents' parameters via policy gradients. Test time visualization provides insight into the learned policies through value function estimates, map tile visitation distribution, and agent-agent dependencies.
An MMO-inspired research game platform for studying emergent behaviors of large populations in a complex environment
{"src": "scitldr-AIC_train", "paper_id": "S1gWz2CcKX"}