Optimization of parameterized policies for reinforcement learning (RL) is an important and challenging problem in artificial intelligence. Among the most common approaches are algorithms based on gradient ascent of a score function representing discounted return. In this paper, we examine the role of these policy gradient and actor-critic algorithms in partially-observable multiagent environments. We show several candidate policy update rules and relate them to a foundation of regret minimization and multiagent learning techniques for the one-shot and tabular cases, leading to previously unknown convergence guarantees. We apply our method to model-free multiagent reinforcement learning in adversarial sequential decision problems (zero-sum imperfect information games), using RL-style function approximation. We evaluate on commonly used benchmark Poker domains, showing performance against fixed policies and empirical convergence to approximate Nash equilibria in self-play with rates similar to or better than a baseline model-free algorithm for zero sum games, without any domain-specific state space reductions.
In this section, we briefly describe the necessary background. While we draw on game-theoretic formalisms, we choose to align our terminology with the RL literature to emphasize the setting and motivations. We include clarifications in Appendix . For details, see @cite_80 @cite_21 .
@cite_80 @cite_21
2112837316 2807340089 2136794369 2127978988
We provide an overview and synthesis of the literatures analyzing games in which players are connected via a network structure. We discuss, in particular, the impact of the structure of the network on individuals’ behaviors. We focus on game theoretic modeling, but also include some discussion of analyses of peer effects, as well as applications to diffusion, employment, crime, industrial organization, and education. We introduce an approach for deep reinforcement learning (RL) that improves upon the efficiency, generalization capacity, and interpretability of conventional approaches through structured perception and relational reasoning. It uses self-attention to iteratively reason about the relations between entities in a scene and to guide a model-free policy. Our results show that in a novel navigation and planning task called Box-World, our agent finds interpretable solutions that improve upon baselines in terms of sample complexity, ability to generalize to more complex scenes than experienced during training, and overall performance. In the StarCraft II Learning Environment, our agent achieves state-of-the-art performance on six mini-games -- surpassing human grandmaster performance on four. By considering architectural inductive biases, our work opens new directions for overcoming important, but stubborn, challenges in deep RL. The author discusses games of both perfect and imperfect information at two levels of structural detail: players' local actions, and their global powers for determining outcomes of the game. Matching logical languages are proposed for both. In particular, at the "action level", imperfect information games naturally model a combined "dynamic-epistemic language"--and correspondences are found between special axioms in this language and particular modes of playing games with their information dynamics. At the "outcome level", the paper presents suitable notions of game equivalence, and some simple representation results. Copyright 2001 by Blackwell Publishing Ltd and the Board of Trustees of the Bulletin of Economic Research This paper introduces a model of ‘theory of mind’, namely, how we represent the intentions and goals of others to optimise our mutual interactions. We draw on ideas from optimum control and game theory to provide a ‘game theory of mind’. First, we consider the representations of goals in terms of value functions that are prescribed by utility or rewards. Critically, the joint value functions and ensuing behaviour are optimised recursively, under the assumption that I represent your value function, your representation of mine, your representation of my representation of yours, and so on ad infinitum. However, if we assume that the degree of recursion is bounded, then players need to estimate the opponent's degree of recursion (i.e., sophistication) to respond optimally. This induces a problem of inferring the opponent's sophistication, given behavioural exchanges. We show it is possible to deduce whether players make inferences about each other and quantify their sophistication on the basis of choices in sequential games. This rests on comparing generative models of choices with, and without, inference. Model comparison is demonstrated using simulated and real data from a ‘stag-hunt’. Finally, we note that exactly the same sophisticated behaviour can be achieved by optimising the utility function itself (through prosocial utility), producing unsophisticated but apparently altruistic agents. This may be relevant ethologically in hierarchal game theory and coevolution.