This paper extends the notion of equilibrium in game theory to learning algorithms in repeated stochastic games. We define a learning equilibrium as an algorithm used by a population of players, such that no player can individually use an alternative algorithm and increase its asymptotic score. We introduce Foolproof Cooperative Learning (FCL), an algorithm that converges to a Tit-for-Tat behavior. It allows cooperative strategies when played against itself while being not exploitable by selfish players. We prove that in repeated symmetric games, this algorithm is a learning equilibrium. We illustrate the behavior of FCL on symmetric matrix and grid games, and its robustness to selfish learners.
Closer to our setting, when the dynamic is unknown, online MARL can extract cooperative solution in some non-cooperative games, and particularly in restricted resource appropriation @cite_6 . Using alternative objectives based on all players reward functions and their propensity to cooperate or defect improves and generalizes the emergence of cooperation in non-cooperative games and limits the risk of being exploited by purely selfish agents @cite_8 .
@cite_6 @cite_8
2963689090 2076900408 2963937357 2153344215
Humanity faces numerous problems of common-pool resource appropriation. This class of multi-agent social dilemma includes the problems of ensuring sustainable use of fresh water, common fisheries, grazing pastures, and irrigation systems. Abstract models of common-pool resource appropriation based on non-cooperative game theory predict that self-interested agents will generally fail to find socially positive equilibria---a phenomenon called the tragedy of the commons. However, in reality, human societies are sometimes able to discover and implement stable cooperative solutions. Decades of behavioral game theory research have sought to uncover aspects of human behavior that make this possible. Most of that work was based on laboratory experiments where participants only make a single choice: how much to appropriate. Recognizing the importance of spatial and temporal resource dynamics, a recent trend has been toward experiments in more complex real-time video game-like environments. However, standard methods of non-cooperative game theory can no longer be used to generate predictions for this case. Here we show that deep reinforcement learning can be used instead. To that end, we study the emergent behavior of groups of independently learning agents in a partially observed Markov game modeling common-pool resource appropriation. Our experiments highlight the importance of trial-and-error learning in common-pool resource appropriation and shed light on the relationship between exclusion, sustainability, and inequality. A cooperative game in characteristic-function form is obtained by allowing a number of individuals to esercise partial control over the constraints of a (generally nonlinear) mathematical programming problem, either directly or through committee voting. Conditions are imposed on the functions defining the programming problem and the control system which suffice to make the game totally balanced. This assures a nonempty core and hence a stable allocation of the full value of the programming problem among the controlling palyers. In the linear case the core is closely related to the solutions of the dual problem. Applications are made to a variety of economic models, including the transferable utility trading economies of Shapley and Shubik and a multishipper one-commodity transshipment model with convex cost functions and concave revenue functions. Dropping the assumption of transferable utility leads to a class of controlled multiobjective or ‘Pareto programming’ problems, which again yield totally balanced games. There has been a resurgence of interest in multiagent reinforcement learning (MARL), due partly to the recent success of deep neural networks. The simplest form of MARL is independent reinforcement learning (InRL), where each agent treats all of its experience as part of its (non stationary) environment. In this paper, we first observe that policies learned using InRL can overfit to the other agents' policies during training, failing to sufficiently generalize during execution. We introduce a new metric, joint-policy correlation, to quantify this effect. We describe a meta-algorithm for general MARL, based on approximate best responses to mixtures of policies generated using deep reinforcement learning, and empirical game theoretic analysis to compute meta-strategies for policy selection. The meta-algorithm generalizes previous algorithms such as InRL, iterated best response, double oracle, and fictitious play. Then, we propose a scalable implementation which reduces the memory requirement using decoupled meta-solvers. Finally, we demonstrate the generality of the resulting policies in three partially observable settings: gridworld coordination problems, emergent language games, and poker. We study a spectrum sharing problem in an unlicensed band where multiple systems coexist and interfere with each other. We first analyze a cooperative setting where all the systems collaborate to achieve a common goal. Under the assumptions that the systems communicate with Gaussian signals and treat interference as noise, we study the structure of the optimal power allocations. We show that any Pareto efficient vector of rates can be achieved with piece-wise constant power allocations. Moreover, if a strong interference condition among all the systems is satisfied, we show that frequency division multiplexing is optimal. We then consider a non-cooperative situation, where the systems act in a selfish and rational way, and investigate how the lack of cooperation can affect performance. Using game theory, we first analyze the possible outcomes of a one shot game, and observe that in many cases an inefficient solution results. We show that by extending the game definition to that of a repeated game, the possibility of building reputations and applying punishments allows to enlarge the set of achievable rates. We present examples that show that in many cases, the performance loss due to lack of cooperation is small. We also provide a converse theorem that proves that our results are tight and quantify the best achievable performance in a non-cooperative scenario