<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In the tutorial, you learned a bit about reinforcement learning and used the `stable-baselines` package to train an agent to beat a random opponent.  In this exercise, you will check your understanding and tinker with the code to deepen your intuition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.game_ai.ex4 import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Set the architecture\n",
    "\n",
    "In the tutorial, you learned one way to design a neural network that can select moves in Connect Four.  The neural network had an output layer with seven nodes: one for each column in the game board.\n",
    "\n",
    "Say now you wanted to create a neural network that can play chess.  How many nodes should you put in the output layer?\n",
    "\n",
    "- Option A: 2 nodes (number of game players)\n",
    "- Option B: 16 nodes (number of game pieces that each player starts with)\n",
    "- Option C: 4672 nodes (number of possible moves)\n",
    "- Option D: 64 nodes (number of squares on the game board)\n",
    "\n",
    "Use your answer to set the value of the `best_option` variable below.  Your answer should be one of `'A'`, `'B'`, `'C'`, or `'D'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill in the blank\n",
    "best_option = ____\n",
    "\n",
    "# Check your answer\n",
    "q_1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%RM_IF(PROD)%%\n",
    "best_option = 'C'\n",
    "q_1.assert_check_passed()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you solution code\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Decide reward\n",
    "\n",
    "In the tutorial, you learned how to give your agent a reward that encourages it to win games of Connect Four.  Consider now training an agent to win at the game [Minesweeper](https://bit.ly/2T5xEY8).  The goal of the game is to clear the board without detonating any bombs.\n",
    "\n",
    "To play this game in Google Search, click on the **[Play]** button at [this link](https://www.google.com/search?q=minesweeper).  \n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/WzoEfKY.png\" width=50%><br/>\n",
    "</center>\n",
    "\n",
    "With each move, one of the following is true:\n",
    "- The agent selected an invalid move (in other words, it tried to uncover a square that was uncovered as part of a previous move).  Let's assume this ends the game, and the agent loses.\n",
    "- The agent clears a square that did not contain a hidden mine.  The agent wins the game, because all squares without mines are revealed.\n",
    "- The agent clears a square that did not contain a hidden mine, but has not yet won or lost the game.\n",
    "- The agent detonates a mine and loses the game.\n",
    "\n",
    "How might you specify the reward for each of these four cases, so that by maximizing the cumulative reward, the agent will try to win the game?\n",
    "\n",
    "After you have decided on your answer, run the code cell below to get credit for completing this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) (Optional) Amend the code\n",
    "\n",
    "In this next part of the exercise, you will amend the code from the tutorial to experiment with creating your own agents!  There are a lot of hyperparameters involved with specifying a reinforcement learning agent, and you'll have a chance to amend them, to see how performance is affected.\n",
    "\n",
    "First, we'll need to make sure that your Kaggle Notebook is set up to run the code.  Begin by looking at the \"Settings\" menu to the right of your notebook.  Your menu will look like one of the following:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/kR1az0y.png\" width=100%><br/>\n",
    "</center>\n",
    "\n",
    "If your \"Internet\" setting appears as a \"Requires phone verification\" link, click on this link.  This will bring you to a new window; then, follow the instructions to verify your account.  After following this step, your \"Internet\" setting will appear \"Off\", as in the example to the right.\n",
    "\n",
    "Once your \"Internet\" setting appears as \"Off\", click to turn it on.  You'll see a pop-up window that you'll need to \"Accept\" in order to complete the process and have the setting switched to \"On\".  Once the Internet is turned \"On\", you're ready to proceed!\n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/gOVh6Aa.png\" width=100%><br/>\n",
    "</center>\n",
    "\n",
    "Begin by running the code cell below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "!pip install 'tensorflow==1.15.0'\n",
    "\n",
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the code cell above outputs `'1.15.0'`, you're ready to proceed!\n",
    "\n",
    "Run the next code cell to create an environment that you'll use to train an agent.  Feel free to make modifications here.  You might like to:\n",
    "- modify the `change_reward()` method to change the rewards that the agent receives in different conditions.  You may also need to modify `self.reward_range` in the `__init__` method (this tuple should always correspond to the minimum and maximum reward that the agent can receive).\n",
    "- change `agent2` to a different agent, like the `\"negamax\"` agent, or a different, custom agent.  Note that the smarter you make the opponent, the harder it will be for your agent to train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kaggle_environments import make, evaluate\n",
    "from gym import spaces\n",
    "\n",
    "class ConnectFourGym:\n",
    "    def __init__(self, agent2=\"random\"):\n",
    "        ks_env = make(\"connectx\", debug=True)\n",
    "        self.env = ks_env.train([None, agent2])\n",
    "        self.rows = ks_env.configuration.rows\n",
    "        self.columns = ks_env.configuration.columns\n",
    "        # Learn about spaces here: http://gym.openai.com/docs/#spaces\n",
    "        self.action_space = spaces.Discrete(self.columns)\n",
    "        self.observation_space = spaces.Box(low=0, high=2, \n",
    "                                            shape=(self.rows,self.columns,1), dtype=np.int)\n",
    "        # Tuple corresponding to the min and max possible rewards\n",
    "        self.reward_range = (-10, 1)\n",
    "        # StableBaselines throws error if these are not defined\n",
    "        self.spec = None\n",
    "        self.metadata = None\n",
    "    def reset(self):\n",
    "        self.obs = self.env.reset()\n",
    "        return np.array(self.obs['board']).reshape(self.rows,self.columns,1)\n",
    "    def change_reward(self, old_reward, done):\n",
    "        if old_reward == 1: # The agent won the game\n",
    "            return 1\n",
    "        elif done: # The opponent won the game\n",
    "            return -1\n",
    "        else: # Reward 1/42\n",
    "            return 1/(self.rows*self.columns)\n",
    "    def step(self, action):\n",
    "        # Check if agent's move is valid\n",
    "        is_valid = (self.obs['board'][int(action)] == 0)\n",
    "        if is_valid: # Play the move\n",
    "            self.obs, old_reward, done, _ = self.env.step(int(action))\n",
    "            reward = self.change_reward(old_reward, done)\n",
    "        else: # End the game and penalize agent\n",
    "            reward, done, _ = -10, True, {}\n",
    "        return np.array(self.obs['board']).reshape(self.rows,self.columns,1), reward, done, _\n",
    "    \n",
    "# Create ConnectFour environment\n",
    "env = ConnectFourGym(agent2=\"random\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continue by running the next code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#$HIDE$\n",
    "!apt-get update\n",
    "!apt-get install -y cmake libopenmpi-dev python3-dev zlib1g-dev\n",
    "!pip install \"stable-baselines[mpi]==2.9.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the cell above outputs `'2.9.0'`, then you can continue!\n",
    "\n",
    "Run the next code cell without changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from stable_baselines.bench import Monitor \n",
    "from stable_baselines.common.vec_env import DummyVecEnv\n",
    "\n",
    "# Create directory for logging training information\n",
    "log_dir = \"log/\"\n",
    "os.makedirs(log_dir, exist_ok=True)\n",
    "\n",
    "# Logging progress\n",
    "monitor_env = Monitor(env, log_dir, allow_early_resets=True)\n",
    "\n",
    "# Create a vectorized environment\n",
    "vec_env = DummyVecEnv([lambda: monitor_env])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next code cell to use the PPO algorithm to train an agent.  You might like to:\n",
    "- change `PPO1` to `A2C` or `DQN` when defining the model in this line of code: `model = PPO1(CustomCnnPolicy, vec_env, verbose=0)`\n",
    "- change the number of filters (in `n_filters`) or filter size (in `filter_size`) in the convolutional layers defined in the `modified_cnn()` function below.  Or, you can change the number of nodes in the final linear layer (in `n_hidden`).  You might also like to add more layers!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stable_baselines import PPO1, A2C, DQN\n",
    "from stable_baselines.a2c.utils import conv, linear, conv_to_fc\n",
    "from stable_baselines.common.policies import CnnPolicy\n",
    "\n",
    "# Neural network for predicting action values\n",
    "def modified_cnn(scaled_images, **kwargs):\n",
    "    activ = tf.nn.relu\n",
    "    layer_1 = activ(conv(scaled_images, 'c1', n_filters=32, filter_size=3, stride=1, \n",
    "                         init_scale=np.sqrt(2), **kwargs))\n",
    "    layer_2 = activ(conv(layer_1, 'c2', n_filters=64, filter_size=3, stride=1, \n",
    "                         init_scale=np.sqrt(2), **kwargs))\n",
    "    layer_2 = conv_to_fc(layer_2)\n",
    "    return activ(linear(layer_2, 'fc1', n_hidden=512, init_scale=np.sqrt(2)))  \n",
    "\n",
    "class CustomCnnPolicy(CnnPolicy):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super(CustomCnnPolicy, self).__init__(*args, **kwargs, cnn_extractor=modified_cnn)\n",
    "        \n",
    "# Initialize agent\n",
    "model = PPO1(CustomCnnPolicy, vec_env, verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, run the code cell to train the agent and view how the rewards evolved during training.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train agent\n",
    "model.learn(total_timesteps=100000)\n",
    "\n",
    "# Plot cumulative reward\n",
    "with open(os.path.join(log_dir, \"monitor.csv\"), 'rt') as fh:    \n",
    "    firstline = fh.readline()\n",
    "    assert firstline[0] == '#'\n",
    "    df = pd.read_csv(fh, index_col=None)['r']\n",
    "df.rolling(window=1000).mean().plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your agent trained well, the plot (which shows average cumulative rewards) should increase over time.\n",
    "\n",
    "If it doesn't look like your agent made enough progress in the 100,000 timesteps below, you might need to increase the number of timesteps for training the model.\n",
    "\n",
    "To watch your trained agent play against an opponent, refer to the tutorial to get the remaining code!"
   ]
  },<commit_msg>fixing ex4 so runs<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In the tutorial, you learned a bit about reinforcement learning and used the `stable-baselines` package to train an agent to beat a random opponent.  In this exercise, you will check your understanding and tinker with the code to deepen your intuition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.game_ai.ex4 import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Set the architecture\n",
    "\n",
    "In the tutorial, you learned one way to design a neural network that can select moves in Connect Four.  The neural network had an output layer with seven nodes: one for each column in the game board.\n",
    "\n",
    "Say now you wanted to create a neural network that can play chess.  How many nodes should you put in the output layer?\n",
    "\n",
    "- Option A: 2 nodes (number of game players)\n",
    "- Option B: 16 nodes (number of game pieces that each player starts with)\n",
    "- Option C: 4672 nodes (number of possible moves)\n",
    "- Option D: 64 nodes (number of squares on the game board)\n",
    "\n",
    "Use your answer to set the value of the `best_option` variable below.  Your answer should be one of `'A'`, `'B'`, `'C'`, or `'D'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill in the blank\n",
    "best_option = ____\n",
    "\n",
    "# Check your answer\n",
    "q_1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%RM_IF(PROD)%%\n",
    "best_option = 'C'\n",
    "q_1.assert_check_passed()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you solution code\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Decide reward\n",
    "\n",
    "In the tutorial, you learned how to give your agent a reward that encourages it to win games of Connect Four.  Consider now training an agent to win at the game [Minesweeper](https://bit.ly/2T5xEY8).  The goal of the game is to clear the board without detonating any bombs.\n",
    "\n",
    "To play this game in Google Search, click on the **[Play]** button at [this link](https://www.google.com/search?q=minesweeper).  \n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/WzoEfKY.png\" width=50%><br/>\n",
    "</center>\n",
    "\n",
    "With each move, one of the following is true:\n",
    "- The agent selected an invalid move (in other words, it tried to uncover a square that was uncovered as part of a previous move).  Let's assume this ends the game, and the agent loses.\n",
    "- The agent clears a square that did not contain a hidden mine.  The agent wins the game, because all squares without mines are revealed.\n",
    "- The agent clears a square that did not contain a hidden mine, but has not yet won or lost the game.\n",
    "- The agent detonates a mine and loses the game.\n",
    "\n",
    "How might you specify the reward for each of these four cases, so that by maximizing the cumulative reward, the agent will try to win the game?\n",
    "\n",
    "After you have decided on your answer, run the code cell below to get credit for completing this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) (Optional) Amend the code\n",
    "\n",
    "In this next part of the exercise, you will amend the code from the tutorial to experiment with creating your own agents!  There are a lot of hyperparameters involved with specifying a reinforcement learning agent, and you'll have a chance to amend them, to see how performance is affected.\n",
    "\n",
    "First, we'll need to make sure that your Kaggle Notebook is set up to run the code.  Begin by looking at the \"Settings\" menu to the right of your notebook.  Your menu will look like one of the following:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/kR1az0y.png\" width=100%><br/>\n",
    "</center>\n",
    "\n",
    "If your \"Internet\" setting appears as a \"Requires phone verification\" link, click on this link.  This will bring you to a new window; then, follow the instructions to verify your account.  After following this step, your \"Internet\" setting will appear \"Off\", as in the example to the right.\n",
    "\n",
    "Once your \"Internet\" setting appears as \"Off\", click to turn it on.  You'll see a pop-up window that you'll need to \"Accept\" in order to complete the process and have the setting switched to \"On\".  Once the Internet is turned \"On\", you're ready to proceed!\n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/gOVh6Aa.png\" width=100%><br/>\n",
    "</center>\n",
    "\n",
    "Begin by running the code cell below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "!pip install 'tensorflow==1.15.0'\n",
    "\n",
    "import tensorflow as tf\n",
    "from kaggle_environments import make, evaluate\n",
    "from gym import spaces\n",
    "\n",
    "!apt-get update\n",
    "!apt-get install -y cmake libopenmpi-dev python3-dev zlib1g-dev\n",
    "!pip install \"stable-baselines[mpi]==2.9.0\"\n",
    "\n",
    "from stable_baselines.bench import Monitor \n",
    "from stable_baselines.common.vec_env import DummyVecEnv\n",
    "from stable_baselines import PPO1, A2C, ACER, ACKTR, TRPO\n",
    "from stable_baselines.a2c.utils import conv, linear, conv_to_fc\n",
    "from stable_baselines.common.policies import CnnPolicy\n",
    "\n",
    "class ConnectFourGym:\n",
    "    def __init__(self, agent2=\"random\"):\n",
    "        ks_env = make(\"connectx\", debug=True)\n",
    "        self.env = ks_env.train([None, agent2])\n",
    "        self.rows = ks_env.configuration.rows\n",
    "        self.columns = ks_env.configuration.columns\n",
    "        # Learn about spaces here: http://gym.openai.com/docs/#spaces\n",
    "        self.action_space = spaces.Discrete(self.columns)\n",
    "        self.observation_space = spaces.Box(low=0, high=2, \n",
    "                                            shape=(self.rows,self.columns,1), dtype=np.int)\n",
    "        # Tuple corresponding to the min and max possible rewards\n",
    "        self.reward_range = (-10, 1)\n",
    "        # StableBaselines throws error if these are not defined\n",
    "        self.spec = None\n",
    "        self.metadata = None\n",
    "    def reset(self):\n",
    "        self.obs = self.env.reset()\n",
    "        return np.array(self.obs['board']).reshape(self.rows,self.columns,1)\n",
    "    def change_reward(self, old_reward, done):\n",
    "        if old_reward == 1: # The agent won the game\n",
    "            return 1\n",
    "        elif done: # The opponent won the game\n",
    "            return -1\n",
    "        else: # Reward 1/42\n",
    "            return 1/(self.rows*self.columns)\n",
    "    def step(self, action):\n",
    "        # Check if agent's move is valid\n",
    "        is_valid = (self.obs['board'][int(action)] == 0)\n",
    "        if is_valid: # Play the move\n",
    "            self.obs, old_reward, done, _ = self.env.step(int(action))\n",
    "            reward = self.change_reward(old_reward, done)\n",
    "        else: # End the game and penalize agent\n",
    "            reward, done, _ = -10, True, {}\n",
    "        return np.array(self.obs['board']).reshape(self.rows,self.columns,1), reward, done, _\n",
    "    \n",
    "# Create ConnectFour environment\n",
    "env = ConnectFourGym(agent2=\"random\")\n",
    "\n",
    "# Create directory for logging training information\n",
    "log_dir = \"log/\"\n",
    "os.makedirs(log_dir, exist_ok=True)\n",
    "\n",
    "# Logging progress\n",
    "monitor_env = Monitor(env, log_dir, allow_early_resets=True)\n",
    "\n",
    "# Create a vectorized environment\n",
    "vec_env = DummyVecEnv([lambda: monitor_env])\n",
    "\n",
    "# Neural network for predicting action values\n",
    "def modified_cnn(scaled_images, **kwargs):\n",
    "    activ = tf.nn.relu\n",
    "    layer_1 = activ(conv(scaled_images, 'c1', n_filters=32, filter_size=3, stride=1, \n",
    "                         init_scale=np.sqrt(2), **kwargs))\n",
    "    layer_2 = activ(conv(layer_1, 'c2', n_filters=64, filter_size=3, stride=1, \n",
    "                         init_scale=np.sqrt(2), **kwargs))\n",
    "    layer_2 = conv_to_fc(layer_2)\n",
    "    return activ(linear(layer_2, 'fc1', n_hidden=512, init_scale=np.sqrt(2)))  \n",
    "\n",
    "class CustomCnnPolicy(CnnPolicy):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super(CustomCnnPolicy, self).__init__(*args, **kwargs, cnn_extractor=modified_cnn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, run the code cell below to train an agent with PPO and view how the rewards evolved during training.  This code is identical to the code from the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize agent\n",
    "model = PPO1(CustomCnnPolicy, vec_env, verbose=0)\n",
    "\n",
    "# Train agent\n",
    "model.learn(total_timesteps=100000)\n",
    "\n",
    "# Plot cumulative reward\n",
    "with open(os.path.join(log_dir, \"monitor.csv\"), 'rt') as fh:    \n",
    "    firstline = fh.readline()\n",
    "    assert firstline[0] == '#'\n",
    "    df = pd.read_csv(fh, index_col=None)['r']\n",
    "df.rolling(window=1000).mean().plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your agent trained well, the plot (which shows average cumulative rewards) should increase over time.\n",
    "\n",
    "Once you have verified that the code runs, try making amendments to see if you can get increased performance.  You might like to:\n",
    "- change `PPO1` to `A2C` (or `ACER` or `ACKTR` or `TRPO`) when defining the model in this line of code: `model = PPO1(CustomCnnPolicy, vec_env, verbose=0)`.  This will let you see how performance can be affected by changing the algorithm from Proximal Policy Optimization [PPO] to one of:\n",
    "  - Advantage Actor-Critic (A2C),\n",
    "  - or Actor-Critic with Experience Replay (ACER),\n",
    "  - Actor Critic using Kronecker-factored Trust Region (ACKTR), or \n",
    "  - Trust Region Policy Optimization (TRPO).\n",
    "- modify the `change_reward()` method in the `ConnectFourGym` class to change the rewards that the agent receives in different conditions.  You may also need to modify `self.reward_range` in the `__init__` method (this tuple should always correspond to the minimum and maximum reward that the agent can receive).\n",
    "- change `agent2` to a different agent when creating the ConnectFour environment with `env = ConnectFourGym(agent2=\"random\")`.  For instance, you might like to use the `\"negamax\"` agent, or a different, custom agent.  Note that the smarter you make the opponent, the harder it will be for your agent to train!"
   ]
  },
{'fraction_non_alphanumeric': 0.15603626530936854, 'fraction_numerical': 0.008986798154922856, 'mean_word_length': 2.723571217056559, 'pattern_counts': {'":': 140, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4204859', 'n_tokens_mistral': 8461, 'n_tokens_neox': 8229, 'n_words': 2653}