Multi-agent cooperation is an important feature of the natural world. Many tasks involve individual incentives that are misaligned with the common good, yet a wide range of organisms from bacteria to insects and humans are able to overcome their differences and collaborate. Therefore, the emergence of cooperative behavior amongst self-interested individuals is an important question for the fields of multi-agent reinforcement learning (MARL) and evolutionary theory. Here, we study a particular class of multi-agent problems called intertemporal social dilemmas (ISDs), where the conflict between the individual and the group is particularly sharp. By combining MARL with appropriately structured natural selection, we demonstrate that individual inductive biases for cooperation can be learned in a model-free way. To achieve this, we introduce an innovative modular architecture for deep reinforcement learning agents which supports multi-level selection. We present results in two challenging environments, and interpret these in the context of cultural and ecological evolution. Nature shows a substantial amount of cooperation at all scales, from microscopic interactions of genomes and bacteria to species-wide societies of insects and humans BID36 . This is in spite of natural selection pushing for short-term individual selfish interests (Darwin, 1859) . In its purest form, altruism can be favored by selection when cooperating individuals preferentially interact with other cooperators, thus realising the rewards of cooperation without being exploited by defectors BID19 BID31 BID9 BID48 BID12 ). However, many other possibilities exist, including kin selection, reciprocity and group selection BID40 Úbeda & Duéñez-Guzmán, 2011; BID52 BID41 BID56 BID50 .Lately the emergence of cooperation among self-interested agents has become an important topic in multi-agent deep reinforcement learning (MARL). and BID25 formalize the problem domain as an intertemporal social dilemma (ISD), which generalizes matrix game social dilemmas to Markov settings. Social dilemmas are characterized by a trade-off between collective welfare and individual utility. As predicted by evolutionary theory, self-interested reinforcement-learning agents are typically unable to achieve the collectively optimal outcome, converging instead to defecting strategies BID45 . The goal is to find multi-agent training regimes in which individuals resolve social dilemmas, i.e., cooperation emerges.Previous work has found several solutions, belonging to three broad categories: 1) opponent modelling BID13 BID31 , 2) long-term planning using perfect knowledge of the game's rules BID33 BID46 ) and 3) a specific intrinsic motivation function drawn from behavioral economics BID25 . These hand-crafted approaches run at odds with more recent end-to-end model-free learning algorithms, which have been shown to have a greater ability to generalize (e.g. BID10 ). We propose that evolution can be applied to remove the hand-crafting of intrinsic motivation, similar to other applications of evolution in deep learning.Evolution has been used to optimize single-agent hyperparameters BID26 , implement black-box optimization BID55 , and to evolve neuroarchitectures BID38 BID51 , regularization BID3 , loss functions BID27 BID24 , behavioral diversity BID6 , and entire reward functions BID49 . These principles tend to be driven by single-agent search and optimization or competitive multi-agent tasks. Therefore there is no guarantee of success when applying them in the ISD setting. More closely related to our domain are evolutionary simulations of predator-prey dynamics BID57 , which used enforced subpopulations to evolve populations of neurons which are sampled to form the hidden layer of a neural network. Real environments don't provide scalar reward signals to learn from. Instead, organisms have developed various internal drives based on either primary or secondary goals BID1 . Here we examined intrinsic rewards based on features derived from other agents in the environment. In accord with evolutionary theory BID0 BID40 , we found that naïvely implementing natural selection via genetic algorithms did not lead to the emergence of cooperation. Furthermore, assortative matchmaking was sufficient to generate cooperative behavior in cases where honest signals were available. Finally, we proposed a new multi-level evolutionary paradigm based on shared reward networks that achieves cooperation in more general situations.Why does evolving intrinsic social preferences promote cooperation? Firstly, evolution ameliorates the intertemporal choice problem by distilling the long timescale of collective fitness into the short timescale of individual reinforcement learning, thereby improving credit assignment between selfish acts and their temporally displaced negative group outcomes BID25 . Secondly, it mitigates the social dilemma itself by allowing evolution to expose social signals that correlate with, for example, an agent's current level of selfishness. Such information powers a range of mechanisms for achieving mutual cooperation like competitive altruism BID21 , other-regarding preferences BID7 , and inequity aversion BID11 . In accord, laboratory experiments show that humans cooperate more readily when they can communicate BID43 BID29 .The shared reward network evolution model was inspired by multi-level selection; yet it does not correspond to the prototypical case of that theory since its lower level units of evolution (the policy networks) are constantly swapping which higher level unit (reward network) they are paired with. Nevertheless , there are a variety of ways in which we see this form of modularity arise in nature. For example , free-living microorganisms occasionally form multi-cellular structures to solve a higher order adaptive problem, like slime mold forming a spore-producing stalk for dispersal BID54 , and many prokaryotes can incorporate plasmids (modules) found in their environment or received from other individuals as functional parts of their genome, thereby achieving cooperation in social dilemmas BID17 BID37 . Alternatively , in humans a reward network may represent a shared "cultural norm", with its fitness based on cultural information accumulated from the groups in which it holds sway. In this way, the spread of norms can occur independently of the success of individual agents BID2 ).For future work , we suggest investigating alternative evolutionary mechanisms for the emergence of cooperation, such as kin selection BID16 and reciprocity BID52 . It would be interesting to see whether these lead to different weights in a reward network, potentially hinting at the evolutionary origins of different social biases. Along these lines, one might consider studying an emergent version of the assortative matchmaking model along the lines suggested by BID22 , adding further generality and power to our setup. Finally, it would be fascinating to determine how an evolutionary approach can be combined with multi-agent communication to produce that most paradoxical of cooperative behaviors: cheap talk.
We introduce a biologically-inspired modular evolutionary algorithm in which deep RL agents learn to cooperate in a difficult multi-agent social game, which could help to explain the evolution of altruism.
{"src": "scitldr-AIC_train", "paper_id": "r1e-nj05FQ"}