<commit_before># Reinforcement Learning Tutorial with Tensorlayer

<br/>

<a href="https://join.slack.com/t/tensorlayer/shared_invite/<KEY>" target="\_blank">
	<div align="center">
		<img src="../../img/join_slack.png" width="40%"/>
	</div>
</a>

<br/>

This repository contains implementation of most popular reinforcement learning algorithms with Tensorlayer 2.0, supporting [Tensorflow 2.0](https://www.tensorflow.org/alpha/guide/effective_tf2). We aim to make the reinforcement learning tutorial for each algorithm simple and straight-forward to use, as this would not only benefits new learners of reinforcement learning, but also provide convenience for senior researchers to testify their new ideas quickly.

## Prerequisites:

* python 3.5
* tensorflow >= 2.0.0 or tensorflow-gpu >= 2.0.0a0
* tensorlayer >= 2.0.1
* tensorflow-probability
* tf-nightly-2.0-preview

*** If you meet the error`AttributeError: module 'tensorflow' has no attribute 'contrib'` when running the code after installing tensorflow-probability, try:

`pip install --upgrade tf-nightly-2.0-preview tfp-nightly`

## Status: Beta

We are currently open to any suggestions or pull requests from you to make the reinforcement learning tutorial with TensorLayer2.0 a better code repository for both new learners and senior researchers. Some of the algorithms mentioned in the this markdown may be not yet available, since we are still trying to implement more RL algorithms and optimize their performances. However, those algorithms listed above will come out in a few weeks, and the repository will keep updating more advanced RL algorithms in the future.

## To Use:

For each tutorial, open a terminal and run:

 `python ***.py --train` for training and `python ***.py --test` for testing.

The tutorial algorithms follow the same basic structure, as shown in file: [`./tutorial_format.py`](https://github.com/tensorlayer/tensorlayer/blob/reinforcement-learning/examples/reinforcement_learning/tutorial_format.py)

## Table of Contents:

| Algorithms      | Observation Space | Action Space | Tutorial Env   |
| --------------- | ----------------- | ------------ | -------------- |
| Q-learning      | Discrete          | Discrete     | FrozenLake     |
| C51             | Discrete          | Discrete     | Pong, CartPole |
| DQN             | Discrete          | Discrete     | FrozenLake     |
| Variants of DQN | Discrete          | Discrete     | Pong, CartPole |
| Retrace         | Discrete          | Discrete     | Pong, CartPole |
| PER             | Discrete          | Discrete     | Pong, CartPole |
| Actor-Critic    | Continuous        | Discrete     | CartPole       |
| A3C             | Continuous        | Continuous   | BipedalWalker  |
| DDPG            | Continuous        | Continuous   | Pendulum       |
| TD3             | Continuous        | Continuous   | Pendulum       |
| SAC             | Continuous        | Continuous   | Pendulum       |
| PG              | Continuous        | Discrete     | CartPole       |
| TRPO            | Continuous        | Continuous   | Pendulum       |
| PPO             | Continuous        | Continuous   | Pendulum       |
| DPPO            | Continuous        | Continuous   | Pendulum       |


## Examples of RL Algorithms:

* **Q-learning**

  Code: `./tutorial_Qlearning.py`

  <u>Paper</u>: [Technical  Note Q-Learning](http://www.gatsby.ucl.ac.uk/~dayan/papers/cjch.pdf)

  <u>Description</u>: 

  ```
  Q-learning is a non-deep-learning method with TD Learning, Off-Policy, e-Greedy Exploration.
  
  Central formula:
  Q(S, A) <- Q(S, A) + alpha * (R + lambda * Q(newS, newA) - Q(S, A))
  
  See David Silver RL Tutorial Lecture 5 - Q-Learning for more details.
  ```

  ​    

* **Deep Q-Network (DQN)**

  <u>Code:</u> `./tutorial_DQN.py`

  <u>Paper</u>: [Human-level control through deep reinforcementlearning](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf)

  [Playing Atari with Deep Reinforcement Learning](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)

  <u>Description</u>: 

  ```
  Deep Q-Network (DQN) is a method of TD Learning, Off-Policy, e-Greedy Exploration (GLIE).
  
  Central formula:
  Q(S, A) <- Q(S, A) + alpha * (R + lambda * Q(newS, newA) - Q(S, A)),
  delta_w = R + lambda * Q(newS, newA).
  
  See David Silver RL Tutorial Lecture 5 - Q-Learning for more details.
  ```

  

* **Double DQN / Dueling DQN / Noisy DQN**

  <u>Code:</u> `./tutorial_DQN_variants.py`

  <u>Paper</u>: [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461)

  <u>Description</u>: 

  ```
  We implement Double DQN, Dueling DQN and Noisy DQN here.
  
  -The max operator in standard DQN uses the same values both to select and to evaluate an action by:
  
     Q(s_t, a_t) = R\_{t+1\} + gamma \* max\_{a}Q\_\{target\}(s_{t+1}, a).
  
  -Double DQN proposes to use following evaluation to address overestimation problem of max operator:
  
     Q(s_t, a_t) = R\_{t+1\} + gamma \* Q\_{target}(s\_\{t+1\}, max{a}Q(s_{t+1}, a)).
  
  -Dueling DQN uses dueling architecture where the value of state and the advantage of each action is estimated separately.
  
  -Noisy DQN propose to explore by adding parameter noises.
  
  
  ```

  


* **Prioritized Experience Replay**

  <u>Code</u>: `./tutorial_prioritized_replay.py`

  <u>Paper</u>: [Prioritized Experience Replay](https://arxiv.org/abs/1511.05952)

  <u>Description:</u>

  ```
  Prioritized experience replay is an efficient replay method that replay important transitions more frequently. Segment tree data structure is used to speed up indexing.
  ```
<commit_msg>change readme<commit_after># Reinforcement Learning Tutorial with Tensorlayer

<br/>

<a href="https://join.slack.com/t/tensorlayer/<KEY>" target="\_blank">
	<div align="center">
		<img src="../../img/join_slack.png" width="40%"/>
	</div>
</a>

<br/>

This repository contains implementation of most popular reinforcement learning algorithms with Tensorlayer 2.0, supporting [Tensorflow 2.0](https://www.tensorflow.org/alpha/guide/effective_tf2). We aim to make the reinforcement learning tutorial for each algorithm simple and straight-forward to use, as this would not only benefits new learners of reinforcement learning, but also provide convenience for senior researchers to testify their new ideas quickly.

## Prerequisites:

* python 3.5
* tensorflow >= 2.0.0 or tensorflow-gpu >= 2.0.0a0
* tensorlayer >= 2.0.1
* tensorflow-probability
* tf-nightly-2.0-preview

*** If you meet the error`AttributeError: module 'tensorflow' has no attribute 'contrib'` when running the code after installing tensorflow-probability, try:

`pip install --upgrade tf-nightly-2.0-preview tfp-nightly`

## Status: Beta

We are currently open to any suggestions or pull requests from you to make the reinforcement learning tutorial with TensorLayer2.0 a better code repository for both new learners and senior researchers. Some of the algorithms mentioned in the this markdown may be not yet available, since we are still trying to implement more RL algorithms and optimize their performances. However, those algorithms listed above will come out in a few weeks, and the repository will keep updating more advanced RL algorithms in the future.

## To Use:

For each tutorial, open a terminal and run:

 `python ***.py --train` for training and `python ***.py --test` for testing.

The tutorial algorithms follow the same basic structure, as shown in file: [`./tutorial_format.py`](https://github.com/tensorlayer/tensorlayer/blob/reinforcement-learning/examples/reinforcement_learning/tutorial_format.py)

## Table of Contents:

| Algorithms      | Observation Space | Action Space | Tutorial Env   |
| --------------- | ----------------- | ------------ | -------------- |
| Q-learning      | Discrete          | Discrete     | FrozenLake     |
| C51             | Discrete          | Discrete     | Pong, CartPole |
| DQN             | Discrete          | Discrete     | FrozenLake     |
| Variants of DQN | Discrete          | Discrete     | Pong, CartPole |
| Retrace         | Discrete          | Discrete     | Pong, CartPole |
| PER             | Discrete          | Discrete     | Pong, CartPole |
| Actor-Critic    | Continuous        | Discrete     | CartPole       |
| A3C             | Continuous        | Continuous   | BipedalWalker  |
| DDPG            | Continuous        | Continuous   | Pendulum       |
| TD3             | Continuous        | Continuous   | Pendulum       |
| SAC             | Continuous        | Continuous   | Pendulum       |
| PG              | Continuous        | Discrete     | CartPole       |
| TRPO            | Continuous        | Continuous   | Pendulum       |
| PPO             | Continuous        | Continuous   | Pendulum       |
| DPPO            | Continuous        | Continuous   | Pendulum       |


## Examples of RL Algorithms:

* **Q-learning**

  Code: `./tutorial_Qlearning.py`

  <u>Paper</u>: [Technical  Note Q-Learning](http://www.gatsby.ucl.ac.uk/~dayan/papers/cjch.pdf)

  <u>Description</u>: 

  ```
  Q-learning is a non-deep-learning method with TD Learning, Off-Policy, e-Greedy Exploration.
  
  Central formula:
  Q(S, A) <- Q(S, A) + alpha * (R + lambda * Q(newS, newA) - Q(S, A))
  
  See David Silver RL Tutorial Lecture 5 - Q-Learning for more details.
  ```

  ​    

* **Deep Q-Network (DQN)**

  <u>Code:</u> `./tutorial_DQN.py`

  <u>Paper</u>: [Human-level control through deep reinforcementlearning](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf)

  [Playing Atari with Deep Reinforcement Learning](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)

  <u>Description</u>: 

  ```
  Deep Q-Network (DQN) is a method of TD Learning, Off-Policy, e-Greedy Exploration (GLIE).
  
  Central formula:
  Q(S, A) <- Q(S, A) + alpha * (R + lambda * Q(newS, newA) - Q(S, A)),
  delta_w = R + lambda * Q(newS, newA).
  
  See David Silver RL Tutorial Lecture 5 - Q-Learning for more details.
  ```

  

* **Double DQN / Dueling DQN / Noisy DQN**

  <u>Code:</u> `./tutorial_DQN_variants.py`

  <u>Paper</u>: [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461)

  <u>Description</u>: 

  ```
  We implement Double DQN, Dueling DQN and Noisy DQN here.
  
  -The max operator in standard DQN uses the same values both to select and to evaluate an action by:
  
     Q(s_t, a_t) = R_{t+1} + gamma * max_{a}Q_{target}(s_{t+1}, a).
  
  -Double DQN proposes to use following evaluation to address overestimation problem of max operator:
  
     Q(s_t, a_t) = R_{t+1} + gamma * Q_{target}(s_{t+1}, max_{a}Q(s_{t+1}, a)).
  
  -Dueling DQN uses dueling architecture where the value of state and the advantage of each action is estimated separately.
  
  -Noisy DQN propose to explore by adding parameter noises.
  
  
  ```

  


* **Prioritized Experience Replay**

  <u>Code</u>: `./tutorial_prioritized_replay.py`

  <u>Paper</u>: [Prioritized Experience Replay](https://arxiv.org/abs/1511.05952)

  <u>Description:</u>

  ```
  Prioritized experience replay is an efficient replay method that replay important transitions more frequently. Segment tree data structure is used to speed up indexing.
  ```

{'fraction_non_alphanumeric': 0.12225482350849115, 'fraction_numerical': 0.00960256068284876, 'mean_word_length': 2.765651155005022, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 69, 'https://': 14, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6426846', 'n_tokens_mistral': 3684, 'n_tokens_neox': 3348, 'n_words': 1097}