Visual attention serves as a means of feature selection mechanism in the perceptual system. Motivated by Broadbent's leaky filter model of selective attention, we evaluate how such mechanism could be implemented and affect the learning process of deep reinforcement learning. We visualize and analyze the feature maps of DQN on a toy problem Catch, and propose an approach to combine visual selective attention with deep reinforcement learning. We experiment with optical flow-based attention and A2C on Atari games. Experiment results show that visual selective attention could lead to improvements in terms of sample efficiency on tested games. An intriguing relation between attention and batch normalization is also discovered.
Several works implement visual attention using the glimpse network in combination with recurrent models @cite_22 @cite_9 . This approach was later extended to deep reinforcement learning in the Atari games domain, either using recurrent models @cite_8 or glimpse sensory @cite_6 . However, these approaches only attempt to integrate visual attention at input level. Therefore, they resembled foveal vision at retinal level rather than visual selective attention, which might also exist in the deeper structure of the network. These pioneered works showed promising results which encourages us to further study how visual attention could benefit deep reinforcement learning.
@cite_9 @cite_22 @cite_6 @cite_8
2951527505 2141399712 2810136571 2195446438
Applying convolutional neural networks to large images is computationally expensive because the amount of computation scales linearly with the number of image pixels. We present a novel recurrent neural network model that is capable of extracting information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution. Like convolutional neural networks, the proposed model has a degree of translation invariance built-in, but the amount of computation it performs can be controlled independently of the input image size. While the model is non-differentiable, it can be trained using reinforcement learning methods to learn task-specific policies. We evaluate our model on several image classification tasks, where it significantly outperforms a convolutional neural network baseline on cluttered images, and on a dynamic visual control problem, where it learns to track a simple object without an explicit training signal for doing so. We describe a model based on a Boltzmann machine with third-order connections that can learn how to accumulate information about a shape over several fixations. The model uses a retina that only has enough high resolution pixels to cover a small area of the image, so it must decide on a sequence of fixations and it must combine the "glimpse" at each fixation with the location of the fixation before integrating the information with information from other glimpses of the same object. We evaluate this model on a synthetic dataset and two image classification datasets, showing that it can perform at least as well as a model trained on whole images. The paper explores the use of reinforcement learning in tasks that involve image-like (but not necessarily raw-pixel) state representations. It proposes a visual attention operator, related to the concept of visual attention presented in previous works. The operator is tested on the game of Pac-Man, where it allows the use of the same agent with various game layouts, no matter what their dimensions (and thus the size of the image-like representation) are. It is shown that the approach is able to summarize information present in the original representation into a fixed-size glimpse and that the approach is able to outperform several more direct approaches. A deep learning approach to reinforcement learning led to a general learner able to train on visual input to play a variety of arcade games at the human and superhuman levels. Its creators at the Google DeepMind's team called the approach: Deep Q-Network (DQN). We present an extension of DQN by "soft" and "hard" attention mechanisms. Tests of the proposed Deep Attention Recurrent Q-Network (DARQN) algorithm on multiple Atari 2600 games show level of performance superior to that of DQN. Moreover, built-in attention mechanisms allow a direct online monitoring of the training process by highlighting the regions of the game screen the agent is focusing on when making decisions.