We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines. Competitive games have been grand challenges for artificial intelligence research since at least the 1950s BID38 BID45 BID6 . In recent years, a number of breakthroughs in AI have been made in these domains by combining deep reinforcement learning (RL) with self-play, achieving superhuman performance at Go and Poker Moravk et al., 2017) . In continuous control domains, competitive games possess a natural curriculum property, as observed in , where complex behaviors have the potential to emerge in simple environments as a result of competition between agents, rather than due to increasing difficulty of manually designed tasks. Challenging collaborative-competitive multi-agent environments have only recently been addressed using end-to-end RL by BID21 , which learns visually complex first-person 2v2 video games to human level. One longstanding challenge in AI has been robot soccer BID23 , including simulated leagues, which has been tackled with machine learning techniques BID37 BID29 but not yet mastered by end-to-end reinforcement learning.We investigate the emergence of co-operative behaviors through multi-agent competitive games. We design a simple research environment with simulated physics in which complexity arises primarily through competition between teams of learning agents. We introduce a challenging multi-agent soccer environment, using MuJoCo BID46 which embeds soccer in a wider universe of possible environments with consistent simulated physics, already used extensively in the machine learning research community BID16 BID5 BID44 . We focus here on multi-agent interaction by using relatively simple bodies with a 3-dimensional action space (though the environment is scalable to more agents and more complex bodies). 1 We use this environment to examine continuous multiagent reinforcement learning and some of its challenges including coordination, use of shaping rewards, exploitability and evaluation.We study a framework for continuous multi-agent RL based on decentralized population-based training (PBT) of independent RL learners BID20 , where individual agents learn off-policy with recurrent memory and decomposed shaping reward channels. In contrast to some recent work where some degree of centralized learning was essential for multi-agent coordinated behaviors (e.g. BID28 BID9 , we demonstrate that end-to-end PBT can lead to emergent cooperative behaviors in our soccer domain. While designing shaping rewards that induce desired cooperative behavior is difficult, PBT provides a mechanism for automatically evolving simple shaping rewards over time, driven directly by competitive match results. We further suggest to decompose reward into separate weighted channels, with individual discount factors and automatically optimize reward weights and corresponding discounts online. We demonstrate that PBT is able to evolve agents' shaping rewards from myopically optimizing dense individual shaping rewards through to focusing relatively more on long-horizon game rewards, i.e. individual agent's rewards automatically align more with the team objective over time. Their behavior correspondingly evolves from random, through simple ball chasing early in the learning process, to more co-operative and strategic behaviors showing awareness of other agents. These behaviors are demonstrated visually and we provide quantitative evidence for coordination using game statistics, analysis of value functions and a new method of analyzing agents' counterfactual policy divergence.Finally, evaluation in competitive multi-agent domains remains largely an open question. Traditionally, multi-agent research in competitive domains relies on handcrafted bots or established human baselines BID21 , but these are often unavailable and difficult to design. In this paper, we highlight that diversity and exploitability of evaluators is an issue, by observing non-transitivities in the agents pairwise rankings using tournaments between trained teams. We apply an evaluation scheme based on Nash averaging BID2 and evaluate our agents based on performance against pre-trained agents in the support set of the Nash average. We have introduced a new 2v2 soccer domain with simulated physics for continuous multi-agent reinforcement learning research, and used competition between agents in this simple domain to train teams of independent RL agents, demonstrating coordinated behavior, including repeated passing motifs. We demonstrated that a framework of distributed population-based-training with continuous control, combined with automatic optimization of shaping reward channels, can learn in this environment end-to-end. We introduced the idea of automatically optimizing separate discount factors for the shaping rewards, to facilitate the transition from myopically optimizing shaping rewards towards alignment with the sparse long-horizon team rewards and corresponding cooperative behavior. We have introduced novel method of counterfactual policy divergence to analyze agent behavior. Our evaluation has highlighted non-transitivities in pairwise match results and the practical need for robustness, which is a topic for future work. Our environment can serve as a platform for multiagent research with continuous physical worlds, and can be easily scaled to more agents and more complex bodies, which we leave for future research.In our soccer environment the reward is invariant over player and we can drop the dependence on i.SVG requires the critic to learn a differentiable Q-function. The true state of the game s and the identity of other agents π \i , are not revealed during a game and so identities must be inferred from their behavior, for example. Further, as noted in BID10 , off-policy replay is not always fully sound in multi-agent environments since the effective dynamics from any single agent's perspective changes as the other agent's policies change. Because of this, we generally model Q as a function of an agents history of observations -typically keeping a low dimensional summary in the internal state of an LSTM: Q π θ (·, ·; ψ) : X × A → R, where X denotes the space of possible histories or internal memory state, parameterized by a neural network with weights ψ. This enables the Q-function to implicitly condition on other players observed behavior and generalize over the diversity of players in the population and diversity of behaviors in replay, Q is learned using trajectory data stored in an experience replay buffer B, by minimizing the k-step return TD-error with off-policy retrace correction BID33 , using a separate target network for bootstrapping, as is also described in BID13 ; . Specifically we minimize: DISPLAYFORM0 where ξ := ((s t , a t , r t )) i+k t=i is a k-step trajectory snippet, where i denotes the timestep of the first state in the snippet, sampled uniformly from the replay buffer B of prior experience, and Q retrace is the off-policy corrected retrace target: DISPLAYFORM1 where, for stability,Q(·, ·;ψ) : X ×A → R andπ are target network and policies BID30 periodically synced with the online action-value critic and policy (in our experiments we sync after every 100 gradient steps), and c s := min(1, π(as|xs) β(as|xs) ), where β denotes the behavior policy which generated the trajectory snippet ξ sampled from B, and i s=i+1 c s := 1. In our soccer experiments k = 40. Though we use off-policy corrections, the replay buffer has a threshold, to ensure that data is relatively recent.When modelling Q using an LSTM the agent's internal memory state at the first timestep of the snippet is stored in replay, along with the trajectory data. When replaying the experience the LSTM is primed with this stored internal state but then updates its own state during replay of the snippet. LSTMs are optimized using backpropagation through time with unrolls truncated to length 40 in our experiments.
We introduce a new MuJoCo soccer environment for continuous multi-agent reinforcement learning research, and show that population-based training of independent reinforcement learners can learn cooperative behaviors
{"src": "scitldr-AIC_train", "paper_id": "BkG8sjR5Km"}