<commit_before># Dopamine

<div align="center">
  <img src="https://google.github.io/dopamine/images/dopamine_logo.png"><br><br>
</div>

Dopamine is a research framework for fast prototyping of reinforcement learning
algorithms. It aims to fill the need for a small, easily grokked codebase in
which users can freely experiment with wild ideas (speculative research).

Our design principles are:

* _Easy experimentation_: Make it easy for new users to run benchmark
                          experiments.
* _Flexible development_: Make it easy for new users to try out research ideas.
* _Compact and reliable_: Provide implementations for a few, battle-tested
                          algorithms.
* _Reproducible_: Facilitate reproducibility in results. In particular, our
                  setup follows the recommendations given by
                  [Machado et al. (2018)][machado].

In the spirit of these principles, this first version focuses on supporting the
state-of-the-art, single-GPU *Rainbow* agent ([Hessel et al., 2018][rainbow])
applied to Atari 2600 game-playing ([Bellemare et al., 2013][ale]).
Specifically, our Rainbow agent implements the three components identified as
most important by [Hessel et al.][rainbow]:

* n-step Bellman updates (see e.g. [Mnih et al., 2016][a3c])
* Prioritized experience replay ([Schaul et al., 2015][prioritized_replay])
* Distributional reinforcement learning ([C51; Bellemare et al., 2017][c51])

For completeness, we also provide an implementation of DQN
([Mnih et al., 2015][dqn]).
For additional details, please see our
[documentation](https://github.com/google/dopamine/tree/master/docs).

This is not an official Google product.

## Instructions
### Install via source
Installing from source allows you to modify the agents and experiments as
you please, and is likely to be the pathway of choice for long-term use.
These instructions assume that you've already set up your favourite package
manager (e.g. `apt` on Ubuntu, `homebrew` on Mac OS X), and that a C++ compiler
is available from the command-line (almost certainly the case if your favourite
package manager works).

The instructions below assume that you will be running Dopamine in a *virtual
environment*. A virtual environment lets you control which dependencies are
installed for which program; however, this step is optional and you may choose
to ignore it.

Dopamine is a Tensorflow-based framework, and we recommend you also consult
the [Tensorflow documentation](https://www.tensorflow.org/install)
for additional details.

Finally, these instructions are for Python 2.7. While Dopamine is Python 3
compatible, there may be some additional steps needed during installation.

#### Ubuntu

First set up the virtual environment:

```
sudo apt-get update && sudo apt-get install virtualenv
virtualenv --python=python2.7 dopamine-env
source dopamine-env/bin/activate
```

This will create a directory called `dopamine-env` in which your virtual
environment lives. The last command activates the environment.

Then, install the dependencies to Dopamine:

```
sudo apt-get update && sudo apt-get install cmake zlib1g-dev
pip install absl-py atari-py gin-config gym opencv-python tensorflow
```

During installation, you may safely ignore the following error message:
*tensorflow 1.10.1 has requirement numpy<=1.14.5,>=1.13.3, but you'll have
numpy 1.15.1 which is incompatible*.

Finally, download the Dopamine source, e.g.

```
git clone https://github.com/google/dopamine.git
```

#### Mac OS X

First set up the virtual environment:

```
pip install virtualenv
virtualenv --python=python2.7 dopamine-env<commit_msg>Update install instructions for tensorflow with GPU support.<commit_after># Dopamine

<div align="center">
  <img src="https://google.github.io/dopamine/images/dopamine_logo.png"><br><br>
</div>

Dopamine is a research framework for fast prototyping of reinforcement learning
algorithms. It aims to fill the need for a small, easily grokked codebase in
which users can freely experiment with wild ideas (speculative research).

Our design principles are:

* _Easy experimentation_: Make it easy for new users to run benchmark
                          experiments.
* _Flexible development_: Make it easy for new users to try out research ideas.
* _Compact and reliable_: Provide implementations for a few, battle-tested
                          algorithms.
* _Reproducible_: Facilitate reproducibility in results. In particular, our
                  setup follows the recommendations given by
                  [Machado et al. (2018)][machado].

In the spirit of these principles, this first version focuses on supporting the
state-of-the-art, single-GPU *Rainbow* agent ([Hessel et al., 2018][rainbow])
applied to Atari 2600 game-playing ([Bellemare et al., 2013][ale]).
Specifically, our Rainbow agent implements the three components identified as
most important by [Hessel et al.][rainbow]:

* n-step Bellman updates (see e.g. [Mnih et al., 2016][a3c])
* Prioritized experience replay ([Schaul et al., 2015][prioritized_replay])
* Distributional reinforcement learning ([C51; Bellemare et al., 2017][c51])

For completeness, we also provide an implementation of DQN
([Mnih et al., 2015][dqn]).
For additional details, please see our
[documentation](https://github.com/google/dopamine/tree/master/docs).

This is not an official Google product.

## Instructions
### Install via source
Installing from source allows you to modify the agents and experiments as
you please, and is likely to be the pathway of choice for long-term use.
These instructions assume that you've already set up your favourite package
manager (e.g. `apt` on Ubuntu, `homebrew` on Mac OS X), and that a C++ compiler
is available from the command-line (almost certainly the case if your favourite
package manager works).

The instructions below assume that you will be running Dopamine in a *virtual
environment*. A virtual environment lets you control which dependencies are
installed for which program; however, this step is optional and you may choose
to ignore it.

Dopamine is a Tensorflow-based framework, and we recommend you also consult
the [Tensorflow documentation](https://www.tensorflow.org/install)
for additional details.

Finally, these instructions are for Python 2.7. While Dopamine is Python 3
compatible, there may be some additional steps needed during installation.

#### Ubuntu

First set up the virtual environment:

```
sudo apt-get update && sudo apt-get install virtualenv
virtualenv --python=python2.7 dopamine-env
source dopamine-env/bin/activate
```

This will create a directory called `dopamine-env` in which your virtual
environment lives. The last command activates the environment.

Then, install the dependencies to Dopamine. If you don't have access to a
GPU, then replace `tensorflow-gpu` with `tensorflow` in the line below
(see [Tensorflow instructions](https://www.tensorflow.org/install/install_linux)
for details).

```
sudo apt-get update && sudo apt-get install cmake zlib1g-dev
pip install absl-py atari-py gin-config gym opencv-python tensorflow-gpu
```

During installation, you may safely ignore the following error message:
*tensorflow 1.10.1 has requirement numpy<=1.14.5,>=1.13.3, but you'll have
numpy 1.15.1 which is incompatible*.

Finally, download the Dopamine source, e.g.

```
git clone https://github.com/google/dopamine.git
```

#### Mac OS X

First set up the virtual environment:

```
pip install virtualenv
virtualenv --python=python2.7 dopamine-env
{'fraction_non_alphanumeric': 0.08088723602071989, 'fraction_numerical': 0.01620401115686014, 'mean_word_length': 4.314043754410727, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '844486', 'n_tokens_mistral': 2181, 'n_tokens_neox': 1965, 'n_words': 932}