<commit_before>---
layout: post 
comments: true
title: Playing Atari games with TensorFlow implementation of Asynchronous one-step Q-Learning
excerpt: "I'll implement and explain an idea of Asynchronous one-step Q-Learning. As an example will be trained an agent to play in classic Atari SpaceInvaders and Pong games, using only raw pixels observations!" 
date: 2016-11-03 22:00:00 
mathjax: true
---

Deep Reinforcement Learning has recently become a really hot area of research, due to the huge amount of breakthroughs in last couple of years. Such "explosion" started by a group of scientists from a start-up company called DeepMind (later it was acquired by Google), who decided to apply current deep learning progress to existing reinforcement learning (RL) approaches. The result paper [Playing Atari with Deep Reinforcement Learning", Mnih et al., 2013](https://arxiv.org/abs/1312.5602) recieves a lot of attention in AI community, since it is the first time, when a single algorithm, using only raw pixels observations, successfully learns how to survive in absolutely different evironments, with different rules and objectives, and in some of the games, it even outperforms human!
Many improvements have been made to Deep Q-Network (DQN) since 2013, few of them: Double DQN, Dueling DQN. In this topic we will implement Google DeepMind's asynchronous one-step Q-Learning method, presented in [Asynchronous Methods for Deep Reinforcement Learning, Mnih et al., 2016.](https://arxiv.org/abs/1602.01783), with classic Atari 2600 games (however it can work with any OpenAI Gym environment with raw visual input).
Although, the main breakthrough of their paper is state-of-the-art policy-based *Asynchronous Advantage Actor-Critic Network (A3C)*, which outperforms value-based Q-Learning methods in both data efficiency and accuracy, it won't be covered in current post.
For implementation was used a deep learning [TensorFlow](http://tensorflow.org) and [Keras](https://keras.io/) libraries.
Code used in this topic can be found at my [github repository](https://github.com/dbobrenko/asynq-learning). All requirements are listed [here](https://github.com/dbobrenko/asynq-learning#requirements).
For impatient, you can download pretrained agent from [TODO](**TODO link to the model**). The model was trained asynchronously in 8 threads over 30 hours on GTX 980 Ti GPU, in total of 30 millions of frames (however it can be trained further).
After model is downloaded and unpacked, you can evaluate it by running:
```python run_dqn.py --logdir 'PATH_TO_DOWNLOADED_FOLDER' --eval```
The resulting videos can be found in `eval/SpaceInvaders-v0/` folder.

<p align="center">
![alt text][gif_trained_pong]
![alt text][gif_trained_spaceinvaders]
  <br><br>
  Figure 1: An illustration of trained agents playing (from left to right): [Pong-v0 OpenAI Gym level](https://gym.openai.com/envs/Pong-v0), [SpaceInvaders-v0 OpenAI Gym level](https://gym.openai.com/envs/SpaceInvaders-v0).
</p>

So let's get started!
<commit_msg>Post layout fixing v4<commit_after>---
layout: post 
comments: true
title: Playing Atari games with TensorFlow implementation of Asynchronous one-step Q-Learning
excerpt: "I'll implement and explain an idea of Asynchronous one-step Q-Learning. As an example will be trained an agent to play in classic Atari SpaceInvaders and Pong games, using only raw pixels observations!" 
date: 2016-11-03 22:00:00 
mathjax: true
---

Deep Reinforcement Learning has recently become a really hot area of research, due to the huge amount of breakthroughs in last couple of years. Such "explosion" started by a group of scientists from a start-up company called DeepMind (later it was acquired by Google), who decided to apply current deep learning progress to existing reinforcement learning (RL) approaches. The result paper [Playing Atari with Deep Reinforcement Learning", Mnih et al., 2013](https://arxiv.org/abs/1312.5602) recieves a lot of attention in AI community, since it is the first time, when a single algorithm, using only raw pixels observations, successfully learns how to survive in absolutely different evironments, with different rules and objectives, and in some of the games, it even outperforms human!
Many improvements have been made to Deep Q-Network (DQN) since 2013, few of them: Double DQN, Dueling DQN. In this topic we will implement Google DeepMind's asynchronous one-step Q-Learning method, presented in [Asynchronous Methods for Deep Reinforcement Learning, Mnih et al., 2016.](https://arxiv.org/abs/1602.01783), with classic Atari 2600 games (however it can work with any OpenAI Gym environment with raw visual input).
Although, the main breakthrough of their paper is state-of-the-art policy-based *Asynchronous Advantage Actor-Critic Network (A3C)*, which outperforms value-based Q-Learning methods in both data efficiency and accuracy, it won't be covered in current post.
For implementation was used a deep learning [TensorFlow](http://tensorflow.org) and [Keras](https://keras.io/) libraries.
Code used in this topic can be found at my [github repository](https://github.com/dbobrenko/asynq-learning). All requirements are listed [here](https://github.com/dbobrenko/asynq-learning#requirements).  
For impatient, you can download pretrained agent from [TODO](**TODO link to the model**). The model was trained asynchronously in 8 threads over 30 hours on GTX 980 Ti GPU, in total of 30 millions of frames (however it can be trained further).
After model is downloaded and unpacked, you can evaluate it by running:
```python run_dqn.py --logdir 'PATH_TO_DOWNLOADED_FOLDER' --eval```
The resulting videos can be found in `eval/SpaceInvaders-v0/` folder.

<p style="text-align: center;">
![alt text][gif_trained_pong]
![alt text][gif_trained_spaceinvaders]
<br><br>
Figure 1: An illustration of trained agents playing (from left to right): [Pong-v0 OpenAI Gym level](https://gym.openai.com/envs/Pong-v0), [SpaceInvaders-v0 OpenAI Gym level](https://gym.openai.com/envs/SpaceInvaders-v0).
</p>

So let's get started!

{'fraction_non_alphanumeric': 0.08075970272502064, 'fraction_numerical': 0.02064409578860446, 'mean_word_length': 5.043912175648702, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '951282', 'n_tokens_mistral': 1797, 'n_tokens_neox': 1605, 'n_words': 790}