Motivated by the difficulty of effecting fundamental change in the architecture of the Internet, in this paper, we study from a theoretical perspective the question of how individuals can join forces toward collective ventures. To that end, we draw on an elementary concept in Internet systems engineering, namely, that of incremental deployability, which we study mathematically and computationally. For example, we show that incremental deployability is at least as general a concept as the Nash equilibrium (in that the latter can be derived from the former). We then draw on this foundation to design and analyze institutional mechanisms that are not only promising to bootstrap emerging Internet architectures but they also have broader applications in social organization beyond its predominant market (and finance)-based character.
The archetypical model to study the evolution of cooperation in society is the prisoner's dilemma. The stag hunt is the archetypical model for the study of coordination. But as @cite_36 notes the models are related: Iterated prisoner's dilemma with an infinite number of stages can assume the form of a stag hunt. The literature on the game-theoretic study of cooperation and coordination (starting with ) has been concerned with the emergence of cooperative phenomena without a centralized entity to facilitate their manifestation. For example, @cite_24 studies how signaling systems can facilitate coordination. In this paper, we study the emergence of cooperation with the assistance of an exterior to the players entity that is able to enforce commitments but our analysis suggests that players may coordination while altogether eschewing commitments.
@cite_36 @cite_24
1575244154 1936969358
Matrix games like Prisoner's Dilemma have guided research on social dilemmas for decades. However, they necessarily treat the choice to cooperate or defect as an atomic action. In real-world social dilemmas these choices are temporally extended. Cooperativeness is a property that applies to policies, not elementary actions. We introduce sequential social dilemmas that share the mixed incentive structure of matrix game social dilemmas but also require agents to learn policies that implement their strategic intentions. We analyze the dynamics of policies learned by multiple self-interested independent learning agents, each using its own deep Q-network, on two Markov games we introduce here: 1. a fruit Gathering game and 2. a Wolfpack hunting game. We characterize how learned behavior in each domain changes as a function of environmental factors including resource abundance. Our experiments show how conflict can emerge from competition over shared resources and shed light on how the sequential nature of real world social dilemmas affects cooperation. Often groups need to meet repeatedly before a decision is reached. Hence, most individual decisions will be contingent on decisions taken previously by others. In particular, the decision to cooperate or not will depend on one’s own assessment of what constitutes a fair group outcome. Making use of a repeated N-person prisoner’s dilemma, we show that reciprocation towards groups opens a window of opportunity for cooperation to thrive, leading populations to engage in dynamics involving both coordination and coexistence, and characterized by cycles of cooperation and defection. Furthermore, we show that this process leads to the emergence of fairness, whose level will depend on the dilemma at stake. Cooperation is a difficult proposition in the face of Darwinian selection. Those that defect have an evolutionary advantage over cooperators who should therefore die out. However, spatial structure enables cooperators to survive through the formation of homogeneous clusters, which is the hallmark of network reciprocity. Here we go beyond this traditional setup and study the spatiotemporal dynamics of cooperation in a population of populations. We use the prisoner's dilemma game as the mathematical model and show that considering several populations simultaneously gives rise to fascinating spatiotemporal dynamics and pattern formation. Even the simplest assumption that strategies between different populations are payoff-neutral with one another results in the spontaneous emergence of cyclic dominance, where defectors of one population become prey of cooperators in the other population, and vice versa. Moreover, if social interactions within different populations are characterized by significantly different temptations to defect, we observe that defectors in the population with the largest temptation counterintuitively vanish the fastest, while cooperators that hang on eventually take over the whole available space. Our results reveal that considering the simultaneous presence of different populations significantly expands the complexity of evolutionary dynamics in structured populations, and it allows us to understand the stability of cooperation under adverse conditions that could never be bridged by network reciprocity alone. In this paper we initiate the study of how collusion alters the quality of solutions obtained in competitive games. The price of anarchy aims to measure the cost of the lack of coordination by comparing the quality of a Nash equilibrium to that of a centrally designed optimal solution. This notion assumes that players act not only selfishly, but also independently. We propose a framework for modeling groups of colluding players, in which members of a coalition cooperate so as to selfishly maximize their collective welfare. Clearly, such coalitions can improve the social welfare of the participants, but they can also harm the welfare of those outside the coalition. One might hope that the improvement for the coalition participants outweighs the negative effects on the others. This would imply that increased cooperation can only improved the overall solution quality of stable outcomes. However, increases in coordination can actually lead to significant decreases in total social welfare. In light of this, we propose the price of collusion as a measure of the possible negative effect of collusion, specifying the factor by which solution quality can deteriorate in the presence of coalitions. We give examples to show that the price of collusion can be arbitrarily high even in convex games. Our main results show that in the context of load-balancing games, the price of collusion depends upon the disparity in market power among the game participants. We show that in some symmetric nonatomic games (where all users have access to the same set of strategies) increased cooperation always improves the solution quality, and in the discrete analogs of such games, the price of collusion is bounded by two. Distributed Artificial Intelligence systems, in which multiple agents interact to improve their individual performance and to enhance the system’s overall utility, are becoming an increasingly pervasive means of conceptualising a diverse range of applications. As the discipline matures, researchers are beginning to strive for the underlying theories and principles which guide the central processes of coordination and cooperation. Here agent communities are modelled using a distributed goal search formalism and it is argued that commitments (pledges to undertake a specified course of action) and conventions (means of monitoring commitments in changing circumstances) are the foundation of coordination in multi-agent systems. An analysis of existing coordination models which use concepts akin to commitments and conventions is undertaken before a new unifying framework is presented. Finally a number of prominent coordination techniques which do not explicitly involve commitments or conventions are reformulated in these terms to demonstrate their compliance with the central hypothesis of this paper. Cooperation in organisms, whether bacteria or primates, has been a difficulty for evolutionary theory since Darwin. On the assumption that interactions between pairs of individuals occur on a probabilistic basis, a model is developed based on the concept of an evolutionarily stable strategy in the context of the Prisoner's Dilemma game. Deductions from the model, and the results of a computer tournament show how cooperation based on reciprocity can get started in an asocial world, can thrive while interacting with a wide range of other strategies, and can resist invasion once fully established. Potential applications include specific aspects of territoriality, mating, and disease. Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do. A key insight is that many, but not all, human individuals have inequity averse social preferences. This promotes a particular resolution of the matrix game social dilemma wherein inequity-averse individuals are personally pro-social and punish defectors. Here we extend this idea to Markov games and show that it promotes cooperation in several types of sequential social dilemma, via a profitable interaction with policy learnability. In particular, we find that inequity aversion improves temporal credit assignment for the important class of intertemporal social dilemmas. These results help explain how large-scale cooperation may emerge and persist. The ad hoc coordination problem is to design an autonomous agent which is able to achieve optimal flexibility and efficiency in a multiagent system with no mechanisms for prior coordination. We conceptualise this problem formally using a game-theoretic model, called the stochastic Bayesian game, in which the behaviour of a player is determined by its private information, or type. Based on this model, we derive a solution, called Harsanyi-Bellman Ad Hoc Coordination (HBA), which utilises the concept of Bayesian Nash equilibrium in a planning procedure to find optimal actions in the sense of Bellman optimal control. We evaluate HBA in a multiagent logistics domain called level-based foraging, showing that it achieves higher flexibility and efficiency than several alternative algorithms. We also report on a human-machine experiment at a public science exhibition in which the human participants played repeated Prisoner's Dilemma and Rock-Paper-Scissors against HBA and alternative algorithms, showing that HBA achieves equal efficiency and a significantly higher welfare and winning rate. 1. The stag hunt Part I. Location: 2. Bargaining with neighbors 3. Stag hunt with neighbors Part II. Signals: 4. Evolution of inference 5. Cheap talk Part III. Association: 6. Choosing partners 7. Coevolution of structure and strategy. The paper considers the repeated prisoner's dilemma in a large-population random-matching setting where players are unable to recognize their opponents. Despite the informational restrictions cooperation is still a sequential equilibrium supported by "contagious" punishments. The equilibrium does not require excessive patience, and contrary to previous thought, need not be extraordinarily fragile. It is robust to the introduction of small amounts of noise and remains nearly efficient. Extensions are discussed to models with heterogeneous rates of time preference and without public randomizations. A key challenge to create a sustainable and energy-efficient society is in making consumer demand adaptive to energy supply, especially renewable supply. In this paper, we propose a partially-centralized organization of consumers, namely, a consumer cooperative for purchasing electricity from the market. We propose a novel multiagent coordination algorithm to shape the energy consumption of the cooperative. In the cooperative, a central coordinator buys the electricity for the whole group and consumers make their own consumption decisions based on their private consumption constraints and preferences. To coordinate individual consumers under incomplete information, we propose an iterative algorithm in which a virtual price signal is sent by the coordinator to induce consumers to shift demand. We prove that our algorithm converges to the central optimal solution. Additionally we analyze the convergence rate of the algorithm via simulations on randomly generated instances. The results indicate scalability with respect to the number of agents and consumption slots. The field of convention emergence studies how agents involved in repeated coordination games can reach consensus through only local interactions. The literature on this topic is vast and is motivated by human societies, mainly addressing coordination problems between human agents, such as who gets to redial after a dropped telephone call. In contrast, real-world engineering problems, such as coordination in wireless sensor networks, involve agents with limited resources and knowledge and thus pose certain restrictions on the complexity of the coordination mechanisms. Due to these restrictions, strategies proposed for human coordination may not be suitable for engineering applications and need to be further explored in the context of real-world application domains. In this article we take the role of designers of large decentralized multi-agent systems. We investigate the factors that speed up the convergence process of agents arranged in different static and dynamic topologies and under different interaction models, typical for engineering applications. We also study coordination problems both under partial observability and in the presence of faults (or noise). The main contributions of this article are that we propose an approach for emergent coordination, motivated by highly constrained devices, such as wireless nodes and swarm bots, in the absence of a central entity and perform extensive theoretical and empirical studies. Our approach is called Win-Stay Lose-probabilistic-Shift, generalizing two well-known strategies in game theory that have been applied in other domains. We demonstrate that our approach performs well in different settings under limited information and imposes minimal system requirements, due to its simplicity. Moreover, our technique outperforms state-of-the-art coordination mechanisms, guarantees full convergence in any topology and has the property that all convention states are absorbing. This paper reports from a field study of a hospital ward and discusses how people achieve coordination through the use of a wide range of interrelated non-digital artifacts, like whiteboards, work schedules, examination sheets, care records, post-it notes etc. These artifacts have multiple roles and functions which in combination facilitate location awareness, continuous coordination, cooperative planning and status overview. We described how actors achieve coordination by using different aspects of these artifacts: their material qualities, the structure they provide as templates and the signs inscribed upon them that are only meaningful to knowledgeable actors. We finally discuss the implication for the design of CSCW tools from the study. Multi-agent systems (MAS) are a field of study of growing interest in a variety of domains such as robotics or distributed controls. The article focuses on decentralized reinforcement learning (RL) in cooperative MAS, where a team of independent learning robots (IL) try to coordinate their individual behavior to reach a coherent joint behavior. We assume that each robot has no information about its teammates' actions. To date, RL approaches for such ILs did not guarantee convergence to the optimal joint policy in scenarios where the coordination is difficult. We report an investigation of existing algorithms for the learning of coordination in cooperative MAS, and suggest a Q-learning extension for ILs, called hysteretic Q-learning. This algorithm does not require any additional communication between robots. Its advantages are showing off and compared to other methods on various applications: bi-matrix games, collaborative ball balancing task and pursuit domain. In this paper, we present Reo, which forms a paradigm for composition of software components based on the notion of mobile channels. Reo is a channel-based exogenous coordination model in which complex coordinators, called connectors, are compositionally built out of simpler ones. The simplest connectors in Reo are a set of channels with well-defined behaviour supplied by users. Reo can be used as a language for coordination of concurrent processes, or as a ‘glue language’ for compositional construction of connectors that orchestrate component instances in a component-based system. The emphasis in Reo is just on connectors and their composition, and not on the entities that connect to, communicate and cooperate through these connectors. Each connector in Reo imposes a specific coordination pattern on the entities (for example, components) that perform I O operations through that connector, without the knowledge of those entities. Channel composition in Reo is a very powerful mechanism for construction of connectors. We demonstrate the expressive power of connector composition in Reo through a number of examples. We show that exogenous coordination patterns that can be expressed as (meta-level) regular expressions over I O operations can be composed in Reo out of a small set of only five primitive channel types. Pro-social punishment and exclusion are common means to elevate the level of cooperation among unrelated individuals. Indeed, it is worth pointing out that the combined use of these two strategies is quite common across human societies. However, it is still not known how a combined strategy where punishment and exclusion are switched can promote cooperation from the theoretical perspective. In this paper, we thus propose two different switching strategies, namely, peer switching that is based on peer punishment and peer exclusion, and pool switching that is based on pool punishment and pool exclusion. Individuals adopting the switching strategy will punish defectors when their numbers are below a threshold and exclude them otherwise. We study how the two switching strategies influence the evolutionary dynamics in the public goods game. We show that an intermediate value of the threshold leads to a stable coexistence of cooperators, defectors, and players adopting the switching strategy in a well-mixed population, and this regardless of whether the pool-based or the peer-based switching strategy is introduced. Moreover, we show that the pure exclusion strategy alone is able to evoke a limit cycle attractor in the evolutionary dynamics, such that cooperation can coexist with other strategies.Pro-social punishment and exclusion are common means to elevate the level of cooperation among unrelated individuals. Indeed, it is worth pointing out that the combined use of these two strategies is quite common across human societies. However, it is still not known how a combined strategy where punishment and exclusion are switched can promote cooperation from the theoretical perspective. In this paper, we thus propose two different switching strategies, namely, peer switching that is based on peer punishment and peer exclusion, and pool switching that is based on pool punishment and pool exclusion. Individuals adopting the switching strategy will punish defectors when their numbers are below a threshold and exclude them otherwise. We study how the two switching strategies influence the evolutionary dynamics in the public goods game. We show that an intermediate value of the threshold leads to a stable coexistence of cooperators, defectors, and players adopting the switching strategy in a well-mixed pop... This paper introduces a model of ‘theory of mind’, namely, how we represent the intentions and goals of others to optimise our mutual interactions. We draw on ideas from optimum control and game theory to provide a ‘game theory of mind’. First, we consider the representations of goals in terms of value functions that are prescribed by utility or rewards. Critically, the joint value functions and ensuing behaviour are optimised recursively, under the assumption that I represent your value function, your representation of mine, your representation of my representation of yours, and so on ad infinitum. However, if we assume that the degree of recursion is bounded, then players need to estimate the opponent's degree of recursion (i.e., sophistication) to respond optimally. This induces a problem of inferring the opponent's sophistication, given behavioural exchanges. We show it is possible to deduce whether players make inferences about each other and quantify their sophistication on the basis of choices in sequential games. This rests on comparing generative models of choices with, and without, inference. Model comparison is demonstrated using simulated and real data from a ‘stag-hunt’. Finally, we note that exactly the same sophisticated behaviour can be achieved by optimising the utility function itself (through prosocial utility), producing unsophisticated but apparently altruistic agents. This may be relevant ethologically in hierarchal game theory and coevolution. This paper is concerned with the game theoretical approach in designing the multiuser downlink beamformers in multicell systems. Sharing the same physical resource, the base-station of each cell wishes to minimize its transmit power subject to a set of target signal-to-interference-plus-noise ratios (SINRs) at the multiple users in the cell. In this context, at first, the paper considers a strategic noncooperative game (SNG) where each base-station greedily determines its optimal downlink beamformer strategy in a distributed manner, without any coordination between the cells. Via the game theory framework, it is shown that this game belongs to the framework of standard functions. The conditions guaranteeing the existence and uniqueness of a Nash Equilibrium (NE) in this competitive design are subsequently examined. The paper then makes a revisit to the fully coordinated design in multicell downlink beamforming, where the optimal beamformers are jointly designed between the base-stations. A comparison between the competitive and coordinated designs shows the benefits of applying the former over the latter in terms of each design's distributed implementation. Finally, in order to improve the efficiency of the NE in the competitive design, the paper considers a more cooperative game through a pricing mechanism. The pricing consideration enables a base-station to steer its beamformers in a more cooperative manner, which ultimately limits the interference induced to other cells. The study on the existence and uniqueness of the new game's NE is then given. The paper also presents a condition on the pricing factors that allow the new NE point to approach the performance established by the coordinated design, while retaining the distributed nature of the multicell game. Behavioral norms are key ingredients that allow agent coordination where societ al laws do not sufficiently constrain agent behaviors. Whereas social laws need to be enforced in a top-down manner, norms evolve in a bottom-up manner and are typically more self-enforcing. While effective norms can significantly enhance performance of individual agents and agent societies, there has been little work in multiagent systems on the formation of social norms. We propose a model that supports the emergence of social norms via learning from interaction experiences. In our model, individual agents repeatedly interact with other agents in the society over instances of a given scenario. Each interaction is framed as a stage game. An agent learns its policy to play the game over repeated interactions with multiple agents. We term this mode of learning social learning, which is distinct from an agent learning from repeated interactions against the same player. We are particularly interested in situations where multiple action combinations yield the same optimal payoff. The key research question is to find out if the entire population learns to converge to a consistent norm. In addition to studying such emergence of social norms among homogeneous learners via social learning, we study the effects of heterogeneous learners, population size, multiple social groups, etc. Humanity faces numerous problems of common-pool resource appropriation. This class of multi-agent social dilemma includes the problems of ensuring sustainable use of fresh water, common fisheries, grazing pastures, and irrigation systems. Abstract models of common-pool resource appropriation based on non-cooperative game theory predict that self-interested agents will generally fail to find socially positive equilibria---a phenomenon called the tragedy of the commons. However, in reality, human societies are sometimes able to discover and implement stable cooperative solutions. Decades of behavioral game theory research have sought to uncover aspects of human behavior that make this possible. Most of that work was based on laboratory experiments where participants only make a single choice: how much to appropriate. Recognizing the importance of spatial and temporal resource dynamics, a recent trend has been toward experiments in more complex real-time video game-like environments. However, standard methods of non-cooperative game theory can no longer be used to generate predictions for this case. Here we show that deep reinforcement learning can be used instead. To that end, we study the emergent behavior of groups of independently learning agents in a partially observed Markov game modeling common-pool resource appropriation. Our experiments highlight the importance of trial-and-error learning in common-pool resource appropriation and shed light on the relationship between exclusion, sustainability, and inequality.