We study a learning dynamic model of routing (congestion) games to explore how an increase in the total demand influences system performance. We focus on non-atomic routing games with two parallel edges of linear cost, where all agents evolve using Multiplicative Weights Updates with a fixed learning rate. Previous game-theoretic equilibrium analysis suggests that system performance is close to optimal in the large population limit, as seen by the Price of Anarchy reduction. In this work, however, we reveal a rather undesirable consequence of non-equilibrium phenomena driven by population increase. As the total demand rises, we prove that the learning dynamics unavoidably become non-equilibrating, typically chaotic. The Price of Anarchy predictions of near-optimal performance no longer apply. To the contrary, the time-average social cost may converge to its worst possible value in the large population limit. Every system has a carrying capacity, above which the dynamics is non-equilibrating. If the equilibrium flow is a symmetric @math split, the system exhibits one period-doubling bifurcation. A single periodic attractor of period two replaces the attracting fixed point when the demand exceeds the carrying capacity. In general, for asymmetric equilibrium flows, increasing the demand destabilizes the system, so that the system eventually becomes Li-Yorke chaotic with positive topological entropy. This demand-driven instability emerges from any pair of linear cost functions. Remarkably, in any non-equilibrating regime, the time-average flows on the edges converge exactly to the equilibrium flows, a property akin to no-regret learning in zero-sum games. Our results extend to any sequence of shrinking learning rates, e.g., @math , by allowing for a dynamically increasing population size.
Game dynamics as physics. Recently, established a robust connection between game theory, online optimization and a ubiquitous class of systems in classical physics known as Hamiltonian dynamics, which naturally exhibit conservation laws. In the case of discrete-time dynamics, such as MWU or gradient descent, the system trajectories are first order approximations of the continuous dynamics; conservation laws as well as recurrence then no longer hold. Instead we get energy" increase and divergence to the boundary, as shown by , as well as volume expansion and Lyapunov chaos in zero-sum games, as shown by . Despite this divergent, chaotic behavior, gradient descent with fixed step size, has vanishing regret in zero-sum games @cite_41 . So far, it is not clear to what extent the connections with Hamiltonian dynamics can be generalized; however, have considered a class of piecewise affine Hamiltonian vector fields whose orbits are piecewise straight lines and developed the connections with best-reply dynamics. The connection between game theory and physics can hopefully enable us to understand and possibly exploit the hidden structure in non-equilibrium game dynamics, similarly to how in this paper we showed formally that chaotic dynamics have their time-average converging to the value equal to the equilibrium.
@cite_41
2962979365 2944311505 2548493877 2848500096
The cornerstone underpinning deep learning is the guarantee that gradient descent on an objective converges to local minima. Unfortunately, this guarantee fails in settings, such as generative adversarial nets, where there are multiple interacting losses. The behavior of gradient-based methods in games is not well understood – and is becoming increasingly important as adversarial and multiobjective architectures proliferate. In this paper, we develop new techniques to understand and control the dynamics in general games. The key result is to decompose the second-order dynamics into two components. The first is related to potential games, which reduce to gradient descent on an implicit function; the second relates to Hamiltonian games, a new class of games that obey a conservation law, akin to conservation laws in classical mechanical systems. The decomposition motivates Symplectic Gradient Adjustment (SGA), a new algorithm for finding stable fixed points in general games. Basic experiments show SGA is competitive with recently proposed algorithms for finding stable fixed points in GANs – whilst at the same time being applicable to – and having guarantees in – much more general games. We show for the first time, to our knowledge, that it is possible to reconcile in online learning in zero-sum games two seemingly contradictory objectives: vanishing time-average regret and non-vanishing step sizes. This phenomenon, that we coin fast and furious" learning in games, sets a new benchmark about what is possible both in max-min optimization as well as in multi-agent systems. Our analysis does not depend on introducing a carefully tailored dynamic. Instead we focus on the most well studied online dynamic, gradient descent. Similarly, we focus on the simplest textbook class of games, two-agent two-strategy zero-sum games, such as Matching Pennies. Even for this simplest of benchmarks the best known bound for total regret, prior to our work, was the trivial one of @math , which is immediately applicable even to a non-learning agent. Based on a tight understanding of the geometry of the non-equilibrating trajectories in the dual space we prove a regret bound of @math matching the well known optimal bound for adaptive step sizes in the online setting. This guarantee holds for all fixed step-sizes without having to know the time horizon in advance and adapt the fixed step-size accordingly. As a corollary, we establish that even with fixed learning rates the time-average of mixed strategies, utilities converge to their exact Nash equilibrium values. We consider in this chapter a class of two-player nonzero-sum stochastic games with incomplete information, which is inspired by recent applications of game theory in network security. We develop fully distributed reinforcement learning algorithms, which require for each player a minimal amount of information regarding the other player. At each time, each player can be in an active mode or in a sleep mode. If †This material is based upon work supported in part by the U.S. Air Force Office of Scientific Research (AFOSR) under grant number AFOSR MURI FA9550-09-1-0249. D R A F T October 2, 2011, 1:59am D R A F T 2 HYBRID LEARNING IN STOCHASTIC GAMES AND ITS APPLICATION IN NETWORK SECURITY a player is in an active mode, she updates her strategy and estimates of unknown quantities using a specific pure or hybrid learning pattern. The players’ intelligence and rationality are captured by the weighted linear combination of different learning patterns. We use stochastic approximation techniques to show that, under appropriate conditions, the pure or hybrid learning schemes with random updates can be studied using their deterministic ordinary differential equation (ODE) counterparts. Convergence to state-independent equilibria is analyzed for special classes of games, namely, games with two actions, and potential games. Results are applied to network security games between an intruder and an administrator, where the noncooperative behaviors are well characterized by the features of distributed hybrid learning. Games generalize the single-objective optimization paradigm by introducing different objective functions for different players. Differentiable games often proceed by simultaneous or alternating gradient updates. In machine learning, games are gaining new importance through formulations like generative adversarial networks (GANs) and actor-critic systems. However, compared to single-objective optimization, game dynamics are more complex and less understood. In this paper, we analyze gradient-based methods with momentum on simple games. We prove that alternating updates are more stable than simultaneous updates. Next, we show both theoretically and empirically that alternating gradient updates with a negative momentum term achieves convergence in a difficult toy adversarial problem, but also on the notoriously difficult to train saturating GANs.