Deep reinforcement learning (RL) has shown impressive results in a variety of domains, learning directly from high-dimensional sensory streams. However, when neural networks are trained in a fixed environment, such as a single level in a video game, they will usually overfit and fail to generalize to new levels. When RL models overfit, even slight modifications to the environment can result in poor agent performance. This paper explores how procedurally generated levels during training can increase generality. We show that for some games procedural level generation enables generalization to new levels within the same distribution. Additionally, it is possible to achieve better performance with less data by manipulating the difficulty of the levels in response to the performance of the agent. The generality of the learned behaviors is also evaluated on a set of human-designed levels. The results suggest that the ability to generalize to human-designed levels highly depends on the design of the level generators. We apply dimensionality reduction and clustering techniques to visualize the generators' distributions of levels and analyze to what degree they can produce levels similar to those designed by a human.
The IMPALA system @cite_10 trains a single network to play multiple games simultaneously. Here, the same games were used for training and testing, and it is in principle possible that the network simply learned individual behaviors for all of these games within the shared model.
@cite_10
2786036274 2548493877 2611421367 2152421692
In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time, which is already a problem in single task learning. We have developed a new distributed agent IMPALA (Importance-Weighted Actor Learner Architecture) that can scale to thousands of machines and achieve a throughput rate of 250,000 frames per second. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace, which was critical for achieving learning stability. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (, 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (, 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents, use less data and crucially exhibits positive transfer between tasks as a result of its multi-task approach. We consider in this chapter a class of two-player nonzero-sum stochastic games with incomplete information, which is inspired by recent applications of game theory in network security. We develop fully distributed reinforcement learning algorithms, which require for each player a minimal amount of information regarding the other player. At each time, each player can be in an active mode or in a sleep mode. If †This material is based upon work supported in part by the U.S. Air Force Office of Scientific Research (AFOSR) under grant number AFOSR MURI FA9550-09-1-0249. D R A F T October 2, 2011, 1:59am D R A F T 2 HYBRID LEARNING IN STOCHASTIC GAMES AND ITS APPLICATION IN NETWORK SECURITY a player is in an active mode, she updates her strategy and estimates of unknown quantities using a specific pure or hybrid learning pattern. The players’ intelligence and rationality are captured by the weighted linear combination of different learning patterns. We use stochastic approximation techniques to show that, under appropriate conditions, the pure or hybrid learning schemes with random updates can be studied using their deterministic ordinary differential equation (ODE) counterparts. Convergence to state-independent equilibria is analyzed for special classes of games, namely, games with two actions, and potential games. Results are applied to network security games between an intruder and an administrator, where the noncooperative behaviors are well characterized by the features of distributed hybrid learning. In this paper, a novel framework is proposed for optimizing the operation and performance of a large-scale multi-hop millimeter wave (mmW) backhaul within a wireless small cell network having multiple mobile network operators (MNOs). The proposed framework enables the small base stations to jointly decide on forming the multi-hop, mmW links over backhaul infrastructure that belongs to multiple, independent MNOs, while properly allocating resources across those links. In this regard, the problem is addressed using a novel framework based on matching theory composed of two, highly inter-related stages: a multi-hop network formation stage and a resource management stage. One unique feature of this framework is that it jointly accounts for both wireless channel characteristics and economic factors during both network formation and resource management. The multi-hop network formation stage is formulated as a one-to-many matching game, which is solved using a novel algorithm, that builds on the so-called deferred acceptance algorithm and is shown to yield a stable and Pareto optimal multi-hop mmW backhaul network. Then, a one-to-many matching game is formulated to enable proper resource allocation across the formed multi-hop network. This game is then shown to exhibit peer effects and, as such, a novel algorithm is developed to find a stable and optimal resource management solution that can properly cope with these peer effects. Simulation results show that, with manageable complexity, the proposed framework yields substantial gains, in terms of the average sum rate, reaching up to 27 and 54 , respectively, compared with a non-cooperative scheme in which inter-operator sharing is not allowed and a random allocation approach. The results also show that our framework improves the statistics of the backhaul sum rate and provides insights on how to manage pricing and the cost of the cooperative mmW backhaul network for the MNOs. We study competition between wireless devices with incomplete information about their opponents. We model such interactions as Bayesian interference games. Each wireless device selects a power profile over the entire available bandwidth to maximize its data rate. Such competitive models represent situations in which several wireless devices share spectrum without any central authority or coordinated protocol. In contrast to games where devices have complete information about their opponents, we consider scenarios where the devices are unaware of the interference they cause to other devices. Such games, which are modeled as Bayesian games, can exhibit significantly different equilibria. We first consider a simple scenario of simultaneous move games, where we show that the unique Bayes-Nash equilibrium is where both devices spread their power equally across the entire bandwidth. We then extend this model to a two-tiered spectrum sharing case where users act sequentially. Here one of the devices, called the primary user, is the owner of the spectrum and it selects its power profile first. The second device (called the secondary user) then responds by choosing a power profile to maximize its Shannon capacity. In such sequential move games, we show that there exist equilibria in which the primary user obtains a higher data rate by using only a part of the bandwidth. In a repeated Bayesian interference game, we show the existence of reputation effects: an informed primary user can bluff to prevent spectrum usage by a secondary user who suffers from lack of information about the channel gains. The resulting equilibrium can be highly inefficient, suggesting that competitive spectrum sharing is highly suboptimal.