<jupyter_start><jupyter_text>Reinforcement Learning on Google Cloud Ran from datalab. You can do this from cloud using `datalab create ...` Or here are steps to get datalab running locally:1. `sudo docker run -it -p "127.0.0.1:8081:8080" -v "${HOME}:/content"      -e "PROJECT_ID=[YOUR_PROJECT_ID]"  gcr.io/cloud-datalab/datalab:local`2. Click user image at top right of datalab, and login to you GCP account. Gives you authentification for everything :)<jupyter_code>%%bash
# Install packages to test model locally.
apt-get update
apt-get install -y python-numpy python-dev cmake zlib1g-dev libjpeg-dev xvfb libav-tools xorg-dev python-opengl libboost-all-dev libsdl2-dev swig libffi-dev
pip install gym
pip install gym[atari]

pip install opencv-python
apt update && apt install -y libsm6 libxext6
apt-get install -y libxrender-dev
pip install keras<jupyter_output>Reading package lists...
Requirement already satisfied: gym in /usr/local/google/home/praneetdutta/.local/lib/python2.7/site-packages
Requirement already satisfied: requests>=2.0 in /usr/local/google/home/praneetdutta/.local/lib/python2.7/site-packages (from gym)
Requirement already satisfied: six in /usr/local/google/home/praneetdutta/.local/lib/python2.7/site-packages (from gym)
Requirement already satisfied: numpy>=1.10.4 in /usr/local/google/home/praneetdutta/.local/lib/python2.7/site-packages (from gym)
Requirement already satisfied: pyglet>=1.2.0 in /usr/local/google/home/praneetdutta/.local/lib/python2.7/site-packages (from gym)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/google/home/praneetdutta/.local/lib/python2.7/site-packages (from requests>=2.0->gym)
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/google/home/praneetdutta/.local/lib/python2.7/site-packages (from requests>=2.0->gym)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /[...]<jupyter_text>First run locally to make sure everything is working.<jupyter_code>%%bash
rm -r rl_on_gcp/my_model
gcloud ml-engine local train \
   --module-name=trainer.trainer \
   --package-path=${PWD}/rl_on_gcp/trainer \
   --\
   --steps=100000\
   --start_train=100\
   --buffer_size=100\
   --save_model=True\
   --model_dir='my_model'<jupyter_output><empty_output><jupyter_text>Run on ML-Engine<jupyter_code>%%bash
JOBNAME=rl_breakout_$(date -u +%y%m%d_%H%M%S)
REGION='us-central1'
BUCKET='dqn-breakout'

gcloud ml-engine jobs submit training $JOBNAME \
        --package-path=$PWD/rl_on_gcp/trainer \
        --module-name=trainer.trainer \
        --region=$REGION \
        --staging-bucket=gs://$BUCKET \
        --scale-tier=BASIC_TPU\
        --runtime-version=1.9 \
        --\
        --steps=5000000\
        --start_train=10000\
        --buffer_size=1000000\
        --save_model=True\
        --model_dir='gs://dqn-breakout/models/'<jupyter_output>jobId: rl_breakout_181207_061313
state: QUEUED<jupyter_text>TODO: why doesn't it use GPU?From logs using `BASIC_GPU`:``` master-replica-0 | Processes:                                                       GPU Memory | I  master-replica-0 master-replica-0 |  GPU       PID   Type   Process name                             Usage      | I  master-replica-0 master-replica-0 |=============================================================================| I  master-replica-0 master-replica-0 |  No running processes found                                                 | I  master-replica-0``` Hyperparameter tuning<jupyter_code>%%writefile hyperparam.yaml
trainingInput:
  scaleTier: BASIC_GPU
  hyperparameters:
    maxTrials: 40
    maxParallelTrials: 5
    enableTrialEarlyStopping: False
    goal: MAXIMIZE    
    hyperparameterMetricTag: reward
    params:
    - parameterName: update_target
      type: INTEGER
      minValue: 500
      maxValue: 5000
      scaleType: UNIT_LOG_SCALE
    - parameterName: init_eta
      type: DOUBLE
      minValue: 0.8
      maxValue: 0.95
      scaleType: UNIT_LOG_SCALE
    - parameterName: learning_rate
      type: DOUBLE
      minValue: 0.00001
      maxValue: 0.001
      scaleType: UNIT_LOG_SCALE
    - parameterName: batch_size
      type: DISCRETE
      discreteValues:
      - 4
      - 16
      - 32
      - 64
      - 128
      - 256
      - 512
%%bash
JOBNAME=rl_breakout_hp_$(date -u +%y%m%d_%H%M%S)
REGION='us-central1'
BUCKET='dqn-breakout'

gcloud ml-engine jobs submit training $JOBNAME \
        --package-path=$PWD/rl_on_gcp/trainer \
        --module-name=trainer.trainer \
        --region=$REGION \
        --staging-bucket=gs://$BUCKET \
        --config=hyperparam.yaml \
        --runtime-version=1.10 \
        --\
        --steps=100000\
        --start_train=10000\
        --buffer_size=10000\
        --model_dir='gs://dqn-breakout/models/hp/'<jupyter_output>jobId: rl_breakout_hp_181128_175711
state: QUEUED<jupyter_text>Launch tensorboardUsing `tensoboard --logdir='gs://dqn-breakout/models/'` or within datalab:<jupyter_code>from google.datalab.ml import TensorBoard as tb

tb.start('gs://crawles-sandbox/rl_on_gcp/hp/4/')
!gsutil ls  gs://crawles-sandbox/rl_on_gcp/hp/4/<jupyter_output>/bin/sh: gsutil: command not found
{'fraction_non_alphanumeric': 0.1357615894039735, 'fraction_numerical': 0.03778730035060382, 'mean_word_length': 2.993001555209953, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '115851', 'license': 'Apache-2.0', 'n_tokens_mistral': 1923, 'n_tokens_neox': 1788, 'n_words': 352, 'path': 'blogs/rl-on-gcp/DQN_Breakout/RL_on_GCP.ipynb', 'repo_name': 'KayvanShah1/training-data-analyst', 'size': '5134'}