A basic question for zero-sum repeated games consists in determining whether the mean payoff per time unit is independent of the initial state. In the special case of "zero-player" games, i.e., of Markov chains equipped with additive functionals, the answer is provided by the mean ergodic theorem. We generalize this result to repeated games. We show that the mean payoff is independent of the initial state for all state-dependent perturbations of the rewards if and only if an ergodicity condition is verified. The latter is characterized by the uniqueness modulo constants of nonlinear harmonic functions (fixed points of the recession function associated to the Shapley operator), or, in the special case of stochastic games with finite action spaces and perfect information, by a reachability condition involving conjugate subsets of states in directed hypergraphs. We show that the ergodicity condition for games only depends on the support of the transition probability, and that it can be checked in polynomial time when the number of states is fixed.
A useful tool to address the issue of the solvability of the ergodic equation , or of the corresponding nonlinear eigenproblem , is the recession function associated with the Shapley operator, which has already been used in several ways @cite_19 @cite_0 @cite_22 . In particular, Rosenberg and Sorin @cite_19 gave conditions for the existence of the mean payoff vector of a two-person zero-sum stochastic game. In their framework, the recession function appears as the Shapley operator of the projective'' game, which corresponds to the game with no running payments.
@cite_0 @cite_19 @cite_22
1788877992 2107101459 1591018827 2084982707
This paper provides an analysis of error propagation in Approximate Dynamic Programming applied to zero-sum two-player Stochastic Games. We provide a novel and unified error propagation analysis in Lp-norm of three well-known algorithms adapted to Stochastic Games (namely Approximate Value Iteration, Approximate Policy Iteration and Approximate Generalized Policy Iteration). We show that we can achieve a stationary policy which is 2γe+e′ (1-γ)2 -optimal, where e is the value function approximation error and e′ is the approximate greedy operator error. In addition, we provide a practical algorithm (AGPI-Q) to solve infinite horizon γ-discounted two-player zero-sum Stochastic Games in a batch setting. It is an extension of the Fitted-Q algorithm (which solves Markov Decisions Processes from data) and can be non-parametric. Finally, we demonstrate experimentally the performance of AGPI-Q on a simultaneous two-player game, namely Alesia. We study quantitatively the effective large-scale behavior of discrete elliptic equations on the lattice ( Z^d ) with random coefficients. The theory of stochastic homogenization relates the random, stationary, and ergodic field of coefficients with a deterministic matrix of effective coefficients. This is done via the corrector problem, which can be viewed as a highly degenerate elliptic equation on the infinite-dimensional space of admissible coefficient fields. In this contribution we develop new quantitative methods for the corrector problem based on the assumption that ergodicity holds in the quantitative form of a Spectral Gap Estimate w.r.t. a Glauber dynamics on coefficient fields—as it is the case for independent and identically distributed coefficients. As a main result we prove an optimal decay in time of the semigroup associated with the corrector problem (i.e. of the generator of the process called “random environment as seen from the particle”). As a corollary we recover existence of stationary correctors (in dimensions (d>2 )) and prove new optimal estimates for regularized versions of the corrector (in dimensions (d 2 )). We also give a self-contained proof of a new estimate on the gradient of the parabolic, variable-coefficient Green’s function, which is a crucial analytic ingredient in our approach. As an application of these results, we prove the first (and optimal) estimates for the approximation of the homogenized coefficients by the popular periodization method in case of independent and identically distributed coefficients. The Koopman operator is a linear but infinite-dimensional operator that governs the evolution of scalar observables defined on the state space of an autonomous dynamical system and is a powerful tool for the analysis and decomposition of nonlinear dynamical systems. In this manuscript, we present a data-driven method for approximating the leading eigenvalues, eigenfunctions, and modes of the Koopman operator. The method requires a data set of snapshot pairs and a dictionary of scalar observables, but does not require explicit governing equations or interaction with a “black box” integrator. We will show that this approach is, in effect, an extension of dynamic mode decomposition (DMD), which has been used to approximate the Koopman eigenvalues and modes. Furthermore, if the data provided to the method are generated by a Markov process instead of a deterministic dynamical system, the algorithm approximates the eigenfunctions of the Kolmogorov backward equation, which could be considered as the “stochastic Koopman operator” (Mezic in Nonlinear Dynamics 41(1–3): 309–325, 2005). Finally, four illustrative examples are presented: two that highlight the quantitative performance of the method when presented with either deterministic or stochastic data and two that show potential applications of the Koopman eigenfunctions. Abstract Given a monotone and homogeneous self-mapping f of the n -dimensional positive cone, a communication matrix M f is introduced and a family F of functions is obtained by multiplying each component of f by arbitrary positive numbers. Assuming that f satisfies a weak form of convexity, a necessary and sufficient criterion on the structure of M f is given so that each function in F has a (nonlinear) eigenvalue. An alternative necessary and sufficient condition in terms of the recession function of f is also provided.