The human ability to coordinate and cooperate has been vital to the development of societies for thousands of years. While it is not fully clear how this behavior arises, social norms are thought to be a key factor in this development. In contrast to laws set by authorities, norms tend to evolve in a bottom-up manner from interactions between members of a society. While much behavior can be explained through the use of social norms, it is difficult to measure the extent to which they shape society as well as how they are affected by other societ al dynamics. In this paper, we discuss the design and evaluation of a reinforcement learning model for understanding how the opportunity to choose who you interact with in a society affects the overall societ al outcome and the strength of social norms. We first study the emergence of norms and then the emergence of cooperation in presence of norms. In our model, agents interact with other agents in a society in the form of repeated matrix-games: coordination games and cooperation games. In particular, in our model, at each each stage, agents are either able to choose a partner to interact with or are forced to interact at random and learn using policy gradients.
While researchers can employ learning strategies in agents to simulate the emergence of norms, it is a difficult task to build simulations that encompass the natural physical and social constraints that individuals are faced within a society. Nevertheless, investigations into normative behavior have been revealing. proposed a replicable multi-agent learning framework for social norm emergence in which agents interact at random in a society @cite_16 . In this simulation, agents had access to actions taken by the opponent but not their identity. They demonstrated that agents trained with a hill-climbing technique, Win-or-Lose-Fast (WoLF) @cite_11 were capable of converging to a norm in a driving simulation that required all agents in a population to drive either on the left side or the right side of the road for maximum societ al gain @cite_16 . In another set of experiments, test the outcome of this framework while constraining agents to interactions with their neighbors @cite_23 . This has since been further extended to agent societies considering a variety of network topologies @cite_4 that have proven suitable for the emergence of norms and conventions as they provide an effective infrastructure in which interactions within a social network can take place.
@cite_16 @cite_4 @cite_23 @cite_11
1630986973 2172193467 92376239 2120327309
Effective norms, emerging from sustained individual interactions over time, can complement societ al rules and significantly enhance performance of individual agents and agent societies. Researchers have used a model that supports the emergence of social norms via learning from interaction experiences where each interaction is viewed as a stage game. In this social learning model, which is distinct from an agent learning from repeated interactions against the same player, an agent learns a policy to play the game from repeated interactions with multiple learning agents. The key research question is to characterize when and how the entire population of homogeneous learners converge to a consistent norm when multiple action combinations yield the same optimal payoff. In this paper we study two extensions to the social learning model that significantly enhances its applicability. We first explore the effects of heterogeneous populations where different agents may be using different learning algorithms. We also investigate norm emergence when agent interactions are physically constrained. We consider agents located on a grid where an agent is more likely to interact with other agents situated closer to it than those that are situated afar. The key new results include the surprising acceleration in learning with limited interaction ranges. We also study the effects of pure-strategy players, i.e., nonlearners in the environment. Social norms play a pivotal role in sustaining social order by regulating individual behaviors in a society. In normative multiagent systems, social norms have been used as an efficient mechanism to govern virtual agent societies towards cooperation and coordination. In this paper, we study the emergence of social norms via learning from repeated local interactions in networked agent societies. We propose a collective learning framework, which imitates the opinion aggregation process in human decision making, to study the impact of agent local collective behaviors on norm emergence in different situations. In the framework, each agent interacts repeatedly with all of its neighbors. At each step, an agent first takes a best-response action towards each of its neighbors and then combines all of these actions into a final action using ensemble learning methods. We conduct extensive experiments to evaluate the framework with respect to different network topologies, learning strategies, numbers of actions, and so on. Experimental results reveal some significant insights into norm emergence in networked agent societies achieved through local collective behaviors. Behavioral norms are key ingredients that allow agent coordination where societ al laws do not sufficiently constrain agent behaviors. Whereas social laws need to be enforced in a top-down manner, norms evolve in a bottom-up manner and are typically more self-enforcing. While effective norms can significantly enhance performance of individual agents and agent societies, there has been little work in multiagent systems on the formation of social norms. We propose a model that supports the emergence of social norms via learning from interaction experiences. In our model, individual agents repeatedly interact with other agents in the society over instances of a given scenario. Each interaction is framed as a stage game. An agent learns its policy to play the game over repeated interactions with multiple agents. We term this mode of learning social learning, which is distinct from an agent learning from repeated interactions against the same player. We are particularly interested in situations where multiple action combinations yield the same optimal payoff. The key research question is to find out if the entire population learns to converge to a consistent norm. In addition to studying such emergence of social norms among homogeneous learners via social learning, we study the effects of heterogeneous learners, population size, multiple social groups, etc. Abstract Learning to act in a multiagent environment is a difficult problem since the normal definition of an optimal policy no longer applies. The optimal policy at any moment depends on the policies of the other agents. This creates a situation of learning a moving target. Previous learning algorithms have one of two shortcomings depending on their approach. They either converge to a policy that may not be optimal against the specific opponents' policies, or they may not converge at all. In this article we examine this learning problem in the framework of stochastic games. We look at a number of previous learning algorithms showing how they fail at one of the above criteria. We then contribute a new reinforcement learning technique using a variable learning rate to overcome these shortcomings. Specifically, we introduce the WoLF principle, “Win or Learn Fast”, for varying the learning rate. We examine this technique theoretically, proving convergence in self-play on a restricted class of iterated matrix games. We also present empirical results on a variety of more general stochastic games, in situations of self-play and otherwise, demonstrating the wide applicability of this method.