Deep reinforcement learning (RL) agents often fail to generalize to unseen environments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images. In this paper, we propose a simple technique to improve a generalization ability of deep RL agents by introducing a randomized (convolutional) neural network that randomly perturbs input observations. It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments. Furthermore, we consider an inference method based on the Monte Carlo approximation to reduce the variance induced by this randomization. We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regularization and data augmentation methods for the same purpose. Deep reinforcement learning (RL) has been applied to various applications, including board games (e.g., Go (Silver et al., 2017) and Chess (Silver et al., 2018) ), video games (e.g., Atari games (Mnih et al., 2015) and StarCraft (Vinyals et al., 2017) ), and complex robotics control tasks (Tobin et al., 2017; Ren et al., 2019) . However, it has been evidenced in recent years that deep RL agents often struggle to generalize to new environments, even when semantically similar to trained agents (Farebrother et al., 2018; Zhang et al., 2018b; Gamrian & Goldberg, 2019; Cobbe et al., 2019) . For example, RL agents that learned a near-optimal policy for training levels in a video game fail to perform accurately in unseen levels (Cobbe et al., 2019) , while a human can seamlessly generalize across similar tasks. Namely, RL agents often overfit to training environments, thus the lack of generalization ability makes them unreliable in several applications, such as health care (Chakraborty & Murphy, 2014) and finance (Deng et al., 2016) . The generalization of RL agents can be characterized by visual changes (Cobbe et al., 2019; Gamrian & Goldberg, 2019) , different dynamics (Packer et al., 2018) , and various structures (Beattie et al., 2016; Wang et al., 2016) . In this paper, we focus on the generalization across tasks where the trained agents take various unseen visual patterns at the test time, e.g., different styles of backgrounds, floors, and other objects (see Figure 1 ). We also found that RL agents completely fail due to small visual changes 1 because it is challenging to learn generalizable representations from high-dimensional input observations, such as images. To improve generalization, several strategies, such as regularization (Farebrother et al., 2018; Zhang et al., 2018b; Cobbe et al., 2019) and data augmentation (Tobin et al., 2017; Ren et al., 2019) , have been proposed in the literature (see Section 2 for further details). In particular, Tobin et al. (2017) showed that training RL agents in various environments generated by randomizing rendering in a simulator improves the generalization performance, leading to a better performance in real environments. This implies that RL agents can learn invariant and robust representations if diverse input observations are provided during training. However, their method is limited by requiring a physics simulator, which may not always be available. This motivates our approach of developing a simple and plausible method applicable to training deep RL agents. The main contribution of this paper is to develop a simple randomization technique for improving the generalization ability across tasks with various unseen visual patterns. Our main idea is to utilize random (convolutional) networks to generate randomized inputs (see Figure 1 (a)), and train RL agents (or their policy) by feeding them into the networks. Specifically, by re-initializing the parameters of random networks at every iteration, the agents are encouraged to be trained under a broad range of perturbed low-level features, e.g., various textures, colors, or shapes. We discover that the proposed idea guides RL agents to learn generalizable features that are more invariant in unseen environments (see Figure 3 ) than conventional regularization (Srivastava et al., 2014; Ioffe & Szegedy, 2015) and data augmentation (Cobbe et al., 2019; Cubuk et al., 2019) techniques. Here, we also provide an inference technique based on the Monte Carlo approximation, which stabilizes the performance by reducing the variance incurred from our randomization method at test time. We demonstrate the effectiveness of the proposed method on the 2D CoinRun (Cobbe et al., 2019) game, the 3D DeepMind Lab exploration task (Beattie et al., 2016) , and the 3D robotics control task (Fan et al., 2018) . For evaluation, the performance of the trained agents is measured in unseen environments with various visual and geometrical patterns (e.g., different styles of backgrounds, objects, and floors), guaranteeing that the trained agents encounter unseen inputs at test time. Note that learning invariant and robust representations against such changes is essential to generalize to unseen environments. In our experiments, the proposed method significantly reduces the generalization gap in unseen environments unlike conventional regularization and data augmentation techniques. For example, compared to the agents learned with the cutout (DeVries & Taylor, 2017 ) data augmentation methods proposed by Cobbe et al. (2019) , our method improves the success rates from 39.8% to 58.7% under 2D CoinRun, the total score from 55.4 to 358.2 for 3D DeepMind Lab, and the total score from 31.3 to 356.8 for the Surreal robotics control task. Our results can be influential to study other generalization domains, such as tasks with different dynamics (Packer et al., 2018) , as well as solving real-world problems, such as sim-to-real transfer (Tobin et al., 2017) . In this paper, we explore generalization in RL where the agent is required to generalize to new environments in unseen visual patterns, but semantically similar. To improve the generalization ability, we propose to randomize the first layer of CNN to perturb low-level features, e.g., various textures, colors, or shapes. Our method encourages agents to learn invariant and robust representations by producing diverse visual input observations. Such invariant features could be useful for several other related topics, like an adversarial defense in RL (see Appendix D for further discussions), sim-toreal transfer (Tobin et al., 2017; Ren et al., 2019) , transfer learning (Parisotto et al., 2016; Rusu et al., 2016a; b) , and online adaptation (Nagabandi et al., 2019) . We provide the more detailed discussions on an extension to the dynamics generalization and failure cases of our method in Appendix L and M, respectively. The adversarial (visually imperceptible) perturbation (Szegedy et al., 2014) to clean input observations can induce the DNN-based policies to generate an incorrect decision at test time (Huang et al., 2017; Lin et al., 2017) . This undesirable property of DNNs has raised major security concerns. In this section, we evaluate if the proposed method can improve the robustness on adversarial Algorithm 1 PPO + random networks, Actor-Critic Style for iteration= 1, 2, · · · do Sample the parameter φ of random networks from prior distribution P (φ) for actor= 1, 2, · · · , N do Run policy π (a|f (s; φ) ; θ) in the given environment for T timesteps Compute advantage estimates end for Optimize L random in equation (3) with respect to θ end for attacks. Our method is expected to improve the robustness against such adversarial attacks because the agents are trained with randomly perturbed inputs. To verify that the proposed method can improve the robustness to adversarial attacks, the adversarial samples are generated using FGSM (Goodfellow et al., 2015) by perturbing inputs to the opposite direction to the most probable action initially predicted by the policy: where ε is the magnitude of noise and a * = arg max a π(a|s; θ) is the action from the policy. Table 4 shows that our proposed method can improve the robustness against FGSM attacks with ε = 0.01, which implies that hidden representations of trained agents are more robust. (2019) , two boxes are painted in the upper-left corner, where their color represents the x-and y-axis velocity to help the agents quickly learn to act optimally. In this way, the agent does not need to memorize previous states, so a simple CNN-based policy without LSTM can effectively perform in our experimental settings. Data augmentation methods. In this paper, we compare a variant of cutout (DeVries & Taylor, 2017) proposed in Cobbe et al. (2019) , grayout, inversion, and color jitter (Cubuk et al., 2019) . Specifically, the cutout augmentation applies a random number of boxes in random size and color to the input, the grayout method averages all three channels of the input, the inversion method inverts pixel values by a 50% chance, and the color jitter changes the characteristics of images commonly used for data augmentation in computer vision tasks: brightness, contrast, and saturation. For every timestep in the cutout augmentation, we first randomly choose the number of boxes from zero to five, assign them a random color and size, and place them in the observation. For the color jitter, the parameters for brightness, contrast, and saturation are randomly chosen in [0.5,1.5]. 10 For each episode, the parameters of these methods are randomized and fixed such that the same image preprocessing is applied within an episode.
We propose a simple randomization technique for improving generalization in deep reinforcement learning across tasks with various unseen visual patterns.
{"src": "scitldr-AIC_train", "paper_id": "HJgcvJBFvB"}