<jupyter_start><jupyter_text>Learning to play Blackjack with Q-learning In this notebook I'll show how Q-learning can be used by an agent with no prior knowledge to teach itself to play Blackjack. While I don't expect the agent to find a way to make money from gambling, I am interested in how artifical intelligence trained over hundreds of thousands of games fares against top human performance. Indeed, by the end of this notebook we will show that a model-free agent can learn to play at near-human levels.To make the comparison we will simulate games under a strategy recommended by a human expert, using the simulation's mean rewards over last 25,000 games as the benchmark. Since mean rewards are expected to be negative in all cases (human or otherwise), we can also reframe this standard RL evaluation metric as something more easily understood to wider audiences: "expected value of funds after 100 games". Using the human strategy, we find that the mean expected value of £100 after 100 games would be £96.65. We will treat this figure as a benchmark for what a "100% accurate" solution looks like.We show that with the help of a grid search and some additional fine-tuning, we can produce an agent whose expected value of £100 after 100 games is £96.39. In other words, **our AI reaches 99.7% of human performance in under an hour**. Special thanks:My implementation relies heavily on the following key sources, which I highly recommend: -  Blackjack environment created by OpenAI Gym  -  Arthur Juliani's Medium tutorial on Q-learning implementation  -   Arthur Juliani's Medium tutorial on policy rule implementation  Contents:1. Create Blackjack environment 2. Q-learning implementation and variations3. Connect Q-learning algorithm with environment4. Training example5. Optimization: training with grid search6. Comparison against human strategy7. Discussion  0) Load modules & basic set-up<jupyter_code>import gym
from gym import spaces
from gym.utils import seeding

import pandas as pd
import numpy as np
import random
import datetime
import time
import pickle
import os

from sklearn.model_selection import ParameterGrid

import matplotlib.pyplot as plt
import seaborn as sns

# hide warnings
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>1) Create Blackjack environment The basics of the environment are handled in local files *blackjack_env.py* and *settings.py*. The former file is the main environment as coded by OpenAI Gym, plus some minor modifications I added to make the game more interesting such as the inclusion of the "Double" action. The latter file contains some parameters I added which influence the laws of the game held in the main environment file, e.g. controlling when the dealer stops taking cards or how many different actions we allow our agent to perform.For those unfamiliar with Blackjack, I will include the description of the game written by OpenAI Gym. I should emphasise that this is a slightly different version to the standard game, since the action "split" is removed to keep things a little simpler.  Blackjack rules: “Blackjack is a card game where the goal is to obtain cards that sum to as near as possible to 21 without going over.  They're playing against a fixed dealer. Face cards (Jack, Queen, King) have point value 10. Aces can either count as 11 or 1, and it's called 'usable' at 11. This game is placed with an infinite deck (or with replacement).The game starts with each (player and dealer) having one face up and one face down card. The player can request additional cards (hit=1) until they decide to stop (stick=0) or exceed 21 (bust). After the player sticks, the dealer reveals their facedown card, and draws until their sum is 17 or greater.  If the dealer goes bust the player wins. If neither player nor dealer busts, the outcome (win, lose, draw) is decided by whose sum is closer to 21.  The reward for winning is +1, drawing is 0, and losing is -1.” – OpenAI Gym source code     NB: The above variant of Blackjack excludes the actions ‘Double’ and ‘Split’, though we reinstate 'Double' to study agents’ approach to risk-taking. We leave out the latter for simplicity reasons.<jupyter_code># load settings of environment from local file
from settings import *

# load the Blackjack environment from local file
from blackjack_env import *

# create environment
env = BlackjackEnv()

# Confirm the environment our agent will play in
ENV_VALS = (BLACKJACK_SCORE, DEALER_STOP, DOUBLE_BIAS, ODDS_BOOST, NUM_ACTIONS)
print("Created environment:")
print("Goal: %i | DealerStop: %i | DoubleBias: %i | DoubleBoost: %i | NumActions: %i" % ENV_VALS)<jupyter_output>Created environment:
Goal: 21 | DealerStop: 4 | DoubleBias: 0 | DoubleBoost: 2 | NumActions: 3<jupyter_text>It is always useful to articulate the environment's state transition matrix before training. This helps us understand the environment our agent needs to traverse in a more mathematical fashion, particularly the probabilities associated with drawing cards randomly. But to keep this notebook focussed on Q-learning itself, I have included that analysis in a separate notebook here.  2) Q-learning implementation and variationsWe will start by implementing a series of selection policies, all of which control how our agent chooses actions. Since the agent will start with a blank canvas, we cannot simply let the agent always pick the "best" action. Rather, these policies help the agent manage the "exploration-exploitation trade-off". In all cases, the agent will start by prioritising "exploration": choosing actions mostly randomly so that it can experience as many different rewards and penalties as possible. As it gains more experience, it will begin to reduce the frequency of these random actions, instead prioritising "exploitation": using its experience so far to choose the action with highest expected future rewards.The policies implemented are:- e-greedy policy- boltzmann policy- both of the above but with Double-Q learning (see more in the following block of code)<jupyter_code>def select_egreedy(Q_row, number_actions, eps=0.9):
    """
    Select an action (index) based on e-greedy policy,
    e.g. random with probability e and per
    current Q-values with probability 1-e
    """
    if np.random.rand() < eps:
        # pick by random choice
        return np.random.choice(np.arange(0, number_actions))   
    else:
        # choose highest expected value action
        return np.argmax(Q_row)    

def select_boltzmann(Q_row, number_actions, tau=1):
    """
    Select an action (index) based on current Q values and
    the current state of annealing, e.g. using inverse
    temperature, tau.
    
    PARAMS:
    tau = inverse temperature, e.g. as tau approaches 0
          boltzmann distribution approaches uniform sampling.
          higher tau prefers actions with higher expected values
    """
    # normalize wrt minimum values, ensuring no negatives
    Q_row_ajd = Q_row - Q_row.min()

    # calculate probabilities under boltzmann distribution
    Q_pr = np.exp(tau * Q_row_ajd) / np.exp(tau * Q_row_ajd).sum()
    
    # Make selection based on probabilities
    try:
        action_value = np.random.choice(Q_pr, p=Q_pr)
    except:
        # we might exceed memory limits for very large numbers
        # when the temperature is very low. In these cases
        # we would do well to simply pick the max value again
        action_value = np.argmax(Q_row)
    return np.argmax(Q_pr == action_value)

def select_a(Q_row, policy="e", param=0.9):
    """ Select action from a row of Q for the given state
    and for a given policy & parameter
    
    if policy == "e" then we will use e-greedy policy
    if policy == "b" then we will use boltzmann distribution
    """
    # choose action based on policy required
    if policy=="e":
        # use e-greedy policy, where param = epsilon
        return select_egreedy(Q_row, env.action_space.n, param)
    else:
        # use boltzmann policy, where param = tau = inverse temperature
        return select_boltzmann(Q_row, env.action_space.n, param)
    
def doubleQ_select(Q_a, Q_b, s, policy="e", policy_param = 0.9):
    """
    Function which enables selection using two Q matrices. <NAME>'s
    original implementation:
       'In our experiments, we calculated the average of the two Q values for each
    action and then performed e-greedy exploration with the resulting average
    Q values.' (<NAME>)
    
    Optional params:
        policy = "e" or "b" for e-greedy and boltzmann respectively
        policy_param = float, epsilon if e-greedy or inverse temperature (tau)
                       if boltzmann 
    """
    
    # take average Q values under consideration for given state
    Q_avg_row =  (Q_a[s] + Q_b[s])/2
    
    # choose action based on given policy and policy parameter
    return select_a(Q_avg_row, policy, policy_param)<jupyter_output><empty_output><jupyter_text>We now move to defining our update rules, which is how the agent learns from experience in playing Blackjack. We will use the standard update rule as well as a variation called **Double Q learning**, where two matrixes are maintained to avoid over-estimation of expected returns. For more information on this variation see https://arxiv.org/abs/1509.06461<jupyter_code>def vanillaQ_update(Q, r, s, s1, a, lr, y):
    """
    Update Q-matrix using standard update rule
    """
    Q[s][a] = Q[s][a] + lr*(r + y*np.max(Q[s1]) - Q[s][a])
    return Q

def doubleQ_update(Q_a, Q_b, r, s0, s1, a,
                   lr, y, prob_update_Q_a = 0.5):
    """
    Randomly select a Q matrix to update, estimating future
    rewards from the other matrix. Per van Hasselt's original
    implementation.
    
    Parameters:
        - prob_update_Q_a: probability of updating the first matrix.
                           (allows experimentation with reliance on
                           the first Q matrix vs the second Q matrix)
    """
    
    if np.random.rand() < prob_update_Q_a:
        # update Q_a matrix, taking copy before update for reference
        Q_old = np.copy(Q_a)
        optimal_a = np.argmax(Q_a[s1][a])
        Q_a[s0][a] = Q_a[s0][a] + lr*(r + y*Q_b[s1][optimal_a] - Q_a[s0][a])
        
    else:
        # update Q_b matrix, taking copy before update for reference
        Q_old = np.copy(Q_b)
        optimal_b = np.argmax(Q_b[s1][a])
        Q_b[s0][a] = Q_b[s0][a] + lr*(r + y*Q_a[s1][optimal_b] - Q_b[s0][a])
        
    return Q_a, Q_b, Q_old<jupyter_output><empty_output><jupyter_text>3) Connect Q-learning algorithm with environmentNow that we have an environment and functions that implement Q-learning, we can connect them together and watch our agent learn to play! We will put this functionality into a function so that we can easily recall it under different parameters in the rest of this notebook.<jupyter_code>def playGame(Q_a=None, Q_b = None, verbose=0,
             dbl_Q=0, lr=0.1, y=0.6,
             policy = "e", policy_param=0.7,
             num_episodes = 1, cb_over_num_eps=10000):
    """
    Main function where agent learns to play Blackjack.
    
    Inputs:
      Q_a, Q_b        = None, None or pre-trained matrix, None
      verbose         = verbosity level
      dbl_Q           = 1 if using double Q learning
      policy          = "e" for e-greedy, "b" for boltzmann
      policy_param    = float controlling above policy
      num_episodes    = number of games to play
      cb_over_num_eps = maintain a callback version of agent 
                        if they have superior performance over
                        this number of games
                        
    Outputs:
      Q_out           = Q-matrix of agent after final episode
      rList           = list of agent's rewards in each episode 
      Q_cb            = Q-matrix of callback agent
      last_cb_ep      = episode number where callback agent was created
    """
    
    # state space needs to be blackjack_score+11
    # e.g. ace on top of a score of 21
    if Q_a is None:
        Q_a = np.zeros([BLACKJACK_SCORE+11,11,2,env.action_space.n])

    # take a copy of Q matrix if we are using double Q learning
    if dbl_Q != 0:
        Q_b = np.copy(Q_a)
    
    # prepare a 'call-back' version of the Q-matrix
    # e.g. the one with best mean rewards over last N episodes
    Q_cb = Q_a.copy()
    best_meanR = -999999    # value holding "best score" yet
    last_cb_ep = 0          # record of last episode the call-back is made to
    a_freq = np.zeros(env.action_space.n)

    # if using Boltzmann policy, set rate of annealing (tau = inverse temperature)
    # grows from start value to 200 over (pc_episodes_tau_growth*num_episodes) steps 
    if policy=="b":
        pc_episodes_tau_growth = 0.6
        end_growth_ep = pc_episodes_tau_growth*num_episodes
        tau_growth = (200/policy_param)**(1/end_growth_ep)

    #create lists to contain total rewards and steps per episode
    rList = []
    for i in range(num_episodes):
        #Reset environment and get first new observation
        s = env.reset()
        rAll = 0
        d = False
        j = 0
        
        # game will not exceed 20 moves, limit while loop out of caution
        while j < 20:
            j+=1

            # choose an acion per given policy        
            if dbl_Q==0:
                Q_row = Q_a[s]
                a = select_a(Q_row, policy, policy_param)  
            else:
                Q_avg_row =  (Q_a[s] + Q_b[s])/2
                a = select_a(Q_avg_row, policy, policy_param) 

            #Get new state and reward from environment
            Q_old = np.copy(Q_a)
            player_hand_old = env.player.copy()
            s1,r,d,_ = env.step(a)
            
            # Update Q-Table with new knowledge, unless in testing mode (lrate = 0.0)
            if lr > 0:
                if dbl_Q==0:
                    Q_a[s][a] = Q_a[s][a] + lr*(r + y*np.max(Q_a[s1]) - Q_a[s][a])
                else:
                    # Double-Q learning: use the second matrix as estimate of expected returns
                    Q_a, Q_b, Q_old = doubleQ_update(Q_a, Q_b, r, s, s1, a, lr, y)
    
            # inspect the step & Q matrix once, at 1/5 through training
            if (i == num_episodes//5) & (verbose > 2):
                if ((dbl_Q==0)|(np.array_equal(Q_b, Q_old))):
                    Q_new = np.copy(Q_a)
                else:
                    Q_new = np.copy(Q_b) 
                show_step(player_hand_old, env.player, env.dealer,
                          s, s1, a, Q_old, Q_new, r, d)
            
            # update rewards and state
            rAll += r
            s = s1

            if d == True:            
                #Reduce chance of random action as we train the model.
                # for e-greedy policy, reduce epsilon
                if policy=="e":
                    if policy_param >= 0.5:
                        policy_param *= 0.99999
                    else:
                        policy_param *= 0.9999
                        
                # for boltzmann policy, increase tau (inverse temperature)
                if policy=="b":
                    if i < end_growth_ep:
                        policy_param *= tau_growth
                
                # For each of the final 5 episodes, show the game for inspection
                if (i >= (num_episodes-5)) & (verbose > 1):
                    show_ep(env.player, env.dealer, r, i)
                
                # we can also reduce the learning rate over time
                # we will only do this after a very large number of games played
                # See last comments in Section 6 where we introduce this idea
                learn_decay_after_ep = 250000
                if (i >= learn_decay_after_ep):
                    lr *= 0.99999
                    
                break

        # append reward to list
        rList.append(rAll)
        
        # update the callback Q-matrix IIF better average over last N episodes
        meanR_lastN_eps = meanR_lastN(rList, n=cb_over_num_eps)
        if ((i > cb_over_num_eps) & (meanR_lastN_eps > best_meanR)):
            
            # take average of Q-values first if double Q learning
            if dbl_Q == 0:
                Q_cb = np.copy(Q_a)
            else:
                Q_cb = (Q_a + Q_b) / 2
            last_cb_ep = i
            best_meanR = meanR_lastN_eps
            
        # show some training feedback every n episodes
        feedback_freq = num_episodes//5
        if verbose > 0:
            if i % feedback_freq == 0:
                print("Ep: {}\tMean reward over last {} episodes: {}. ({}-policy @ {})".format(i,
                        cb_over_num_eps, meanR_lastN_eps,
                        policy, np.around(policy_param, decimals=5)))
        
    # show overall success history of agent 
    if verbose > 1:
        plt.figure(figsize=(12,4))
        plt.plot(np.cumsum(rList))
        plt.title('Performance vs. episodes')
        plt.ylabel('Total Returns')
        plt.xlabel('Episode number')
        plt.axvline(x=last_cb_ep, linestyle="--", color="black")    # show the last callback episode
        plt.show()
        
    # before returning Q matrix, take average if double Q-learning
    if dbl_Q == 0:
        Q_out = np.copy(Q_a)
    else:
        Q_out = (Q_a + Q_b)/2
        
    return Q_out, rList, Q_cb, last_cb_ep<jupyter_output><empty_output><jupyter_text>4) Training example Before we start training, we want to have some functions that let us evaluate the agent's performance numerically, as well as some ability to visually inspect what is going on at both a high- and granular-level.<jupyter_code>def mode(l):
    """ Find the mode of a given list l. """
    return max(set(l), key=l.count)

def meanR_lastN(r_list, n=200):
    """ Find mean over the last n items in a list """
    return np.mean(r_list[-n:])

def expected_value_funds(meanR, funds_in=100, games_played=100):
    """ Returns the expected value of funds after n games, given the meanR
        of an associated strategy """
    return funds_in + (games_played * meanR)

def calc_success_rate2(rList, eval_last_n=50000, produce_ts=False, title_add=" (final agent)"):
    """ Calcuate success rate over moving window using a given rList
        NB: This version takes double odds into account"""
    # filter to just positive and negative lists
    wins_list = np.array(rList)
    loss_list = wins_list.copy()
    wins_list[wins_list < 0] = 0
    loss_list[loss_list > 0] = 0
    ts_success = []
      
    # calculate average success rate for evaluation period
    num_wins = np.sum(wins_list)
    num_losses = abs(np.sum(loss_list))
    mean_success_rate = num_wins / (num_wins + num_losses)
    
    if produce_ts:
        # calculate success rate over moving window
        window=eval_last_n 
        ts_step=1  
        for i in np.arange(window, len(rList), ts_step):
            num_wins = np.sum(wins_list[i-window:i])
            num_losses = abs(np.sum(loss_list[i-window:i]))
            success_in_window = num_wins / (num_wins + num_losses)
            ts_success.append(success_in_window)
            
        # plot success over time, plus mean success rate in eval period
        plt.figure()
        plt.suptitle("Success rate over time{}".format(title_add))
        plt.title("Mean success rate during evaluation: %.3f" % mean_success_rate)
        plt.plot(ts_success)
        plt.ylim([0.0,1.0])
        plt.ylabel("Success rate")
        plt.xlabel("Number of episodes after initial evaluation @ Ep. {}".format(window))
        plt.show()

    return ts_success, mean_success_rate

def calc_success_rate(rList, eval_last_n=50000, produce_ts=False, title_add=" (final agent)"):
    """ Calcuate success rate over moving window using a given rList.
        NB: This version ignores the impact of double odds games. """
    # convert to binary 1/0 for success/failure
    r_01 = [1*(r>0) for r in rList]
    ts_success = []
    
    # calculate average success rate for evaluation period
    num_wins = np.sum(r_01)
    num_losses = np.sum([1*(r<0) for r in rList])
    mean_success_rate = num_wins / (num_wins + num_losses)
    
    if produce_ts:
        # calculate success rate over moving window
        window=eval_last_n 
        ts_step=1
        for i in np.arange(window, len(rList), ts_step):
            num_wins = np.sum(r_01[i-window:i])
            num_losses = np.sum([1*(r<0) for r in rList[i-window:i]])
            success_in_window = num_wins / (num_wins + num_losses)
            ts_success.append(success_in_window)
            
        # plot success over time, plus mean success rate in eval period
        plt.figure()
        plt.suptitle("Success rate over time{}".format(title_add))
        plt.title("Mean success rate during evaluation: %.3f" % mean_success_rate)
        plt.plot(ts_success)
        plt.ylim([0.0,1.0])
        plt.ylabel("Success rate")
        plt.xlabel("Number of episodes after initial evaluation @ Ep. {}".format(window))
        plt.show()

    return ts_success, mean_success_rate

def convert_first_1_to_11(l):
    """ Convert the first 1 in a hand to 11, e.g. to faciliate understanding of hands played involving an Ace """
    for idx, i in enumerate(l):
        if i == 1:
            l[idx] = 11
            break
    return l
def show_ep(player_hand, dealer_hand, reward, ep_no, cumulative=True):
    """ Summarise the episode in text once it is complete """
    
    # set some placeholder strings to indicate who won/lost/drew
    pl_bust = ""
    dl_bust = ""
    
    # show if anyone went bust with XXX
    if is_bust(player_hand):
        pl_bust = "XXX"
    if is_bust(dealer_hand):
        dl_bust = "XXX"
    
    # show who won, using !!! for win and --- for draw
    if reward > 0:
        pl_win = "!!!"
        dl_win = ""
    elif reward < 0:
        pl_win = ""
        dl_win = "!!!"
    else:
        pl_win = "---"
        dl_win = "---"
        
    # update hands to show effects of an Ace being used as 11 (instead of 1)
    if usable_ace(player_hand):
        player_hand = convert_first_1_to_11(player_hand)
    if usable_ace(dealer_hand):
        dealer_hand = convert_first_1_to_11(dealer_hand)
        
    # display results
    print("--- SUMMARY OF EP #{} ---".format(ep_no))
    if cumulative:
        # simplest view
        print("Player:\t{} {}{}".format(np.cumsum(player_hand), pl_bust, pl_win))
        print("Dealer:\t{} {}{}".format(np.cumsum(dealer_hand), dl_bust, dl_win))
        print(player_hand)
        print(dealer_hand)
    else:
        # more detailed view, showing progression of cards from initial (visible) hands to final hands
        print("Player:\t{} >> {} >> {} {}{}".format(player_hand[:2], player_hand, sum_hand(player_hand),
                                                    pl_bust, pl_win))
        print("Dealer:\t{} >> {} >> {} {}{}".format(dealer_hand[:1], dealer_hand, sum_hand(dealer_hand),
                                                    dl_bust, dl_win))

def show_step(player_hand_old, player_hand_new, dealer_hand,
              s0, s1, a, Q0, Q1, r, d):
    """ Give a detailed view of each step in a hand """
    # current state summary and action
    print("""-----------------------
Player's old hand:\t{} (={})
Dealer's face-up card:\t{}
Player had useable:\t{}
Player action:\t\t{} ({})""".format(player_hand_old, s0[0], s0[1], s0[2], a, action_dict(a)))
    
    # effects of action
    if player_hand_old != player_hand_new:
        print("""Player got card:\t{}""".format(player_hand_new[-1]))
        
    # Dealer's eventual outcome
    print("Dealer's hand:\t{} (={})".format(dealer_hand, sum_hand(dealer_hand)))
    print("{}\n{}".format(s0, s1))
    
    print("r:{} | Done: {}".format(r, d))
    
    # show two Q matrices separately
    inspect_Q(Q0)
    inspect_Q(Q1)
    
    # show the difference
    Q_diff = Q1 - Q0
    inspect_Q(Q_diff)

def inspect_Q(Q_in):
    """ Inspect the Q matrix as heatmaps
    Split out each action's Q-values for all possible states"""
        
    # Heatmaps: Show side-by-side action Q-matrix, with two rows for each Usable Ace state
    color_limits = np.max([abs(np.min(Q_in)), np.max(Q_in)])
    i_labels = ['No useable ace', 'Useable ace']
    j_labels = ['Stick', 'Hit', 'Double', 'Surrender', 'Invest'][:NUM_ACTIONS]
    
    # new row of plots for each of the binary states Ace / no Ace
    for i in np.arange(0, len(i_labels)):    
        plt.figure()
        f, axes = plt.subplots(1,len(j_labels), sharex=False, sharey=False,
                                          figsize=(12,6))
        plt.title(i_labels[i])
        # show Q values for each possible action, across all hand values x initial dealer card
        for j in np.arange(0, len(j_labels)):
            # show each heat map, adding color bar for the last plot only
            showCbar = j == len(j_labels)-1
            # obtain the relevant Q values - hiding non-used states (4- & 22+ for player / 0 for dealer)
            g = sns.heatmap(Q_in[4:22,1:,i,j], cmap='seismic_r', cbar=showCbar,
                            vmin = -color_limits, vmax = color_limits, center=0, ax=axes[j])
            g.set_title("{}\n - {} - ".format(j_labels[j], i_labels[i]))
            g.set_xlabel('Dealer initial card')
            # set agent hand value range of 4-21 inclusive, dealer 2-11 inclusive
            g.set_yticklabels(np.arange(4,22), rotation=0)
            g.set_xticklabels(np.arange(1,11), rotation=0)  # where Ace = 1 for now
            # only show y label on first plot
            if j == 0:
                g.set_ylabel('Agent hand value')
                
        plt.show()<jupyter_output><empty_output><jupyter_text>We are almost ready to start training. All we have to do first is choose how long we want to train the agent for, how many games we are going to evaluate it over, and of course some hyper-parameter values. We will start with some parameter values which seem sensible, and move onto more robust experimentation in the following section. For this reason we will keep the number of training episodes and evaluation window fairly low for now.<jupyter_code>### Set scope of training and evaluation
# number of episodes to train agent over
NUM_EPISODES_TRAIN = 15000           

# evaluate agent with mean rewards over last (N) episodes
EVAL_LAST_N_EPS = 3000                 

### Set verbosity level
# 0: nothing
# 1: validation metrics only
# 2: validation, perf-vs-episodes, last 5 games
# 3: inspect Q-matrix update 1/5 of way into training
VERBOSE =2
# Set hyper-parameters: grid format will be beneficial when we optimize these 
grid = ParameterGrid({"dbl_Q" : [0],                # 1 to use Double Q learning
                          "lr": [0.01],             # learning rate
                          "y": [0.25],              # gamma
                          "policy": "b",            # "e" for e-greedy, "b" for boltzmann
                          "policy_param":  [1],     # epsilon / tau (inverse temperature)
                          "num_episodes": [NUM_EPISODES_TRAIN],
                          "cb_over_num_eps": [EVAL_LAST_N_EPS],
                          "verbose": [VERBOSE]})<jupyter_output><empty_output><jupyter_text>Finally, we can start training an agent in the next block, wrapping the learning process, evaluation & record management into a function we can call again later. This should complete in less than 5 seconds - quite impressive for 15000 games!<jupyter_code>def train_agent(p_idx, params, exportMe=True, eval_metrics=[]):
    # confirm parameters used for training
    agent_summary = (params['dbl_Q'], params['lr'], params['y'],
                     params['policy'], params['policy_param'])
    print("---------------------------\nNow training agent {}/{}".format(p_idx+1, grid_size))
    print("DblQ: %.2f | LR: %.3f | g: %.3f | pol: %s | pol_param: %.5f" % agent_summary)
    
    # train agent using current parameters
    start_time = time.time()
    Q, rList, Q_CB, lastCB_ep = playGame(Q_a=None, Q_b=None, **params)
    end_time = time.time()
    time_elapsed = end_time - start_time  # time difference in seconds
    games_per_sec = NUM_EPISODES_TRAIN / time_elapsed 
    print("...completed in %.1f seconds @ %.1f games per second" % (time_elapsed,
                                                                    games_per_sec))
    
    # calculate evaluation metrics
    eval_final = meanR_lastN(rList, n=EVAL_LAST_N_EPS)
    eval_cb = meanR_lastN(rList[:lastCB_ep], n=EVAL_LAST_N_EPS)
    
    # calculate the success rate metrics
    show_success_ts_plot = VERBOSE > 1
    ts_success_final, eval_final_success = calc_success_rate2(rList, eval_last_n=EVAL_LAST_N_EPS,
                                                             produce_ts = show_success_ts_plot,
                                                             title_add=" (final agent)")
    ts_success_cb, eval_cb_success = calc_success_rate2(rList[:lastCB_ep], eval_last_n=EVAL_LAST_N_EPS,
                                                              produce_ts = show_success_ts_plot,
                                                              title_add=" (CB agent)")

    if VERBOSE>-1:
        print("---------------------------\nValidation metrics\n---------------------------")
        print("After final episode, mean reward of {} over last {} episodes --> success rate: {}".format(eval_final, EVAL_LAST_N_EPS, eval_final_success))
        print("After episode {}, mean reward of {} over last {} episodes --> success rate: {}".format(lastCB_ep, eval_cb, EVAL_LAST_N_EPS, eval_cb_success))
    
    # Pickle Q matrices for later use
    name_fi = ""
    name_cb = ""
    if exportMe:
        print("---------------------------\nPickling agents")
        # pickle the final agent
        export_name = 'Qmatrix_expID_{}_Fi_DblQ{}_lr{}_g{}_pol{}_polparam{}.p'.format(p_idx, params['dbl_Q'], params['lr'],
                                                                    params['y'], params['policy'], params['policy_param'])
        name_fi = Q_DATA_DIR + export_name
        pickle.dump(Q , open(name_fi, "wb" ) )

        # Pickle the call back too, referencing the call-back episode in filename
        export_name_cb = 'Qmatrix_expID_{}_CB{}_DblQ{}_lr{}_g{}_pol{}_polparam{}.p'.format(p_idx, lastCB_ep,
                                                                                  params['dbl_Q'], params['lr'],
                                                                                  params['y'], params['policy'], params['policy_param'])
        name_cb = Q_DATA_DIR + export_name_cb 
        pickle.dump(Q_CB, open(name_cb, "wb" ) )

        # pickle the rLists using for later analysis (same filenames but different folder)
        pickle.dump(rList , open(RLIST_DATA_DIR + export_name, "wb" ) )
        pickle.dump(rList[:lastCB_ep], open(RLIST_DATA_DIR + export_name_cb, "wb" ) )

        # form Vector of validation metrics, for both final agent and call-back agent
        eval_vec_fi = [name_fi, params['dbl_Q'], params['lr'], params['y'],
                       params['policy'], params['policy_param'],
                       params['num_episodes'], params['cb_over_num_eps'],
                       0, eval_final, eval_final_success]
        eval_vec_cb = [name_cb, params['dbl_Q'], params['lr'], params['y'],
                       params['policy'], params['policy_param'],
                       params['num_episodes'], params['cb_over_num_eps'],
                       lastCB_ep, eval_cb, eval_cb_success]
        eval_metrics.append(eval_vec_fi)
        eval_metrics.append(eval_vec_cb)
        
    # return metrics and filepaths for stored Q matrices
    return eval_metrics, name_fi, name_cb<jupyter_output><empty_output><jupyter_text>As shown in the graph detailing performance vs episodes, the agent has clearly learned a strategy around 6000 games in which is several times better than the actions it was taking initially.  The summary of five games played between episode 14,995 and 14,999 shows some slightly curious actions, such as sticking at a hand value of 12 in episode 14,497. At least our agent wins the final game, achieving a hand value of 19 versus the dealer's 18.In any case, we see that the Q-learning process is working, and we are ready to move on to longer training sessions.<jupyter_code># just one set of parameters to go through at this stage
grid_size = len(grid)
all_eval_metrics, _, _ = train_agent(0, grid[0], False)<jupyter_output>---------------------------
Now training agent 1/1
DblQ: 0.00 | LR: 0.010 | g: 0.250 | pol: b | pol_param: 1.00000
Ep: 0	Mean reward over last 3000 episodes: -1.0. (b-policy @ 1.00059)
Ep: 3000	Mean reward over last 3000 episodes: -0.518. (b-policy @ 5.85148)
Ep: 6000	Mean reward over last 3000 episodes: -0.24666666666666667. (b-policy @ 34.21966)
Ep: 9000	Mean reward over last 3000 episodes: -0.102. (b-policy @ 200.0)
Ep: 12000	Mean reward over last 3000 episodes: -0.09866666666666667. (b-policy @ 200.0)
--- SUMMARY OF EP #14995 ---
Player:	[ 5 11 14] 
Dealer:	[ 3  4  5 15 21] !!!
[5, 6, 3]
[3, 1, 1, 10, 6]
--- SUMMARY OF EP #14996 ---
Player:	[ 5 15 19] 
Dealer:	[10 21] !!!
[5, 10, 4]
[10, 11]
--- SUMMARY OF EP #14997 ---
Player:	[ 1  6 12] 
Dealer:	[ 2 12 14 17] !!!
[1, 5, 6]
[2, 10, 2, 3]
--- SUMMARY OF EP #14998 ---
Player:	[ 7 17] 
Dealer:	[ 4 14 15 21] !!!
[7, 10]
[4, 10, 1, 6]
--- SUMMARY OF EP #14999 ---
Player:	[ 2 11 19] !!!
Dealer:	[10 18] 
[2, 9, 8]
[10, 8]<jupyter_text>5) Optimization : training with grid searchNow that we know the Q-learning process is working, we can expand the grid search and simply-reuse our earlier functions to optimize learning. We will keep the grid fairly small so that this notebook does not get too large. (The first time I performed the grid search was with a much larger grid in Spyder, which I prefer for larger tasks.)We will alo raise the number of episodes to train and evaluate over, so that we can get more robust estimations of performance. If we were carrying out analysis in Spyder we should raise the numbers even higher.At this point it is also useful to reduce verbosity to just validation metrics. We will split this into two chunks to reduce risk of any errors / crashes, with the first chunk using only the Boltzmann policy and the second only an e-greedy policy.Warning: The next two blocks could take up to an hour to complete and may be difficult to interrupt!<jupyter_code># number of episodes to train agent over
NUM_EPISODES_TRAIN = 125000

# evaluate agent with mean rewards over last (N) episodes
EVAL_LAST_N_EPS = 25000 
### BOLTZMANN AGENTS
# reduce verbosity level - feedback validation metrics only
VERBOSE = 0

# Create a grid of hyper parameters to search over
grid = ParameterGrid({"dbl_Q" : [0, 1],                 # 1 to use Double Q learning
                          "lr": [0.01, 0.001],          # learning rate
                          "y": [0.25, 0.5],             # gamma
                          "policy": ["b"],              # "e" for e-greedy, "b" for boltzmann
                          "policy_param":  [1],         # tau (inverse temperature)
                          "num_episodes": [NUM_EPISODES_TRAIN],
                          "cb_over_num_eps": [EVAL_LAST_N_EPS],
                          "verbose": [VERBOSE]})
grid_size = len(grid)

# create a new array to store evaluation metrics for all agents
# we pass this to the function so that it can append a new row
# on each iteration
all_eval_metrics = []
for p_idx, params in enumerate(grid):
    all_eval_metrics, _, _ = train_agent(p_idx, params, True, all_eval_metrics)
### E-GREEDY AGENTS
# Create a grid of hyper parameters to search over
grid = ParameterGrid({"dbl_Q" : [0, 1],                 # 1 to use Double Q learning
                          "lr": [0.01, 0.001],          # learning rate
                          "y": [0.25, 0.5],             # gamma
                          "policy": ["e"],              # "e" for e-greedy, "b" for boltzmann ## MODIFIED ##
                          "policy_param":  [0.6],       # epsilon ## MODIFIED ##
                          "num_episodes": [NUM_EPISODES_TRAIN],
                          "cb_over_num_eps": [EVAL_LAST_N_EPS],
                          "verbose": [VERBOSE]})
grid_size = len(grid)

# pass the same evaluation metric array
for p_idx, params in enumerate(grid):
    all_eval_metrics, _, _ = train_agent(p_idx, params, True, all_eval_metrics)<jupyter_output>---------------------------
Now training agent 1/8
DblQ: 0.00 | LR: 0.010 | g: 0.250 | pol: e | pol_param: 0.60000
...completed in 186.9 seconds @ 668.9 games per second
---------------------------
Validation metrics
---------------------------
After final episode, mean reward of -0.0696 over last 25000 episodes --> success rate: 0.43433475077140354
After episode 109355, mean reward of -0.05652 over last 25000 episodes --> success rate: 0.4310176975765794
---------------------------
Pickling agents
---------------------------
Now training agent 2/8
DblQ: 0.00 | LR: 0.010 | g: 0.500 | pol: e | pol_param: 0.60000
...completed in 240.9 seconds @ 518.8 games per second
---------------------------
Validation metrics
---------------------------
After final episode, mean reward of -0.0562 over last 25000 episodes --> success rate: 0.43257759290579145
After episode 123462, mean reward of -0.05072 over last 25000 episodes --> success rate: 0.43242618504592245
---------------------------
Picklin[...]<jupyter_text>We can now summarise the results for all our trained agents and export to a CSV. A quick inspection of the top 5 and bottom 5 performances suggests that the higher learning rate of 0.01 and lower gamma of 0.25 are preferable.<jupyter_code># define column names and form a dataframe of results
# we order by the evaluation metric MeanR_lastN and show top 5
col_names = ['agent_filepath', 'DoubleQ', 'lr', 'y', 'policy', 'pol_param', 'num_episodes',
             'cb_over_num_episodes', 'cb_creation_idx', 'meanR_lastN', 'success_rate_lastN']

eval_df = pd.DataFrame(data=np.array(all_eval_metrics), columns=col_names) \
            .sort_values(by='meanR_lastN', ascending=True) \
            .reset_index().drop('index', 1)

# export to csv
filename = 'validation_results/results.csv'
eval_df.to_csv(filename)
print("Exported results to {}".format(filename))
# show top 5
eval_df.head()
# show bottom 5
eval_df.tail()<jupyter_output><empty_output><jupyter_text>6. Comparison against human strategyUnsurprisingly, none of our agents discovered a way to make money out of Blackjack in the long-run. We know this since their average reward per game maxes out around -0.04 (rounded down from -0.03676 for the very best agent). In other words, if the best agent started with £100, then after 100 games at £1 per play they would likely leave with only £96.An interesting comparison we can make is whether this £96 retained is better than what a human could achieve. This is a little difficult to calculate precisely since our version of Blackjack is non-standard and excludes the "split" option, which can be helpful in hedging bets. That said, BlackjackApprenticeship.com offers advice for non-splitting scenarios, which appears to be a sensible set of instructions we can use in our version of the game.We have encoded the advice into matrices held in csv format, visualised below:<jupyter_code>def action_dict(a, short=False):
    """ Return the initial letter or name of action given an action index"""
    if short:
        lookup = {0: "S", 1: "H", 2: "D", 3: "U", 4: "I"}
    else:
        lookup = {0: "Stick", 1: "Hit", 2: "Double", 3: "Surrender", 4: "Invest"}
    return lookup[a]

def make_adviceChart(Q_in, add_title='', smeChart=False):
    """ Find the argmax for each value of Q matrix along the action dimension
        Convert each to the name label of the action & color code 
        This acts as an easy 'advice chart' to support understanding 
        
        smeChart = flag indicating this chart conveys Subject Matter Experise
                   e.g. uses a human strategy rather than AI strategy"""
    # take top-expectation action per state
    if not(smeChart):
        Qa_max = np.argmax(Q_in, axis=-1)
        
    # split into no usable ace / useable ace states
    f, axes = plt.subplots(1,2, sharex=False, sharey=False,
                                      figsize=(12,6))
    plt.title('Advised action chart per state')
    j_labels = ['No usable Ace', 'Usable Ace']
    for j in [0,1]:
        # load human advice charts rather than Q matrix if requested
        if ((smeChart) & (j==0)):
            Qa_max = np.array(pd.read_csv('additional_analysis/sme_advice_noUseableAce.csv'))
            Qa_maxJ = Qa_max[4:,:]
        elif ((smeChart) & (j==1)):
            Qa_max = np.array(pd.read_csv('additional_analysis/sme_advice_UseableAce.csv'))
            Qa_maxJ = Qa_max[4:,:]
        else:
            Qa_maxJ = Qa_max[4:22,1:,j]
            
        # convert action IDs to labels
        Qa_maxJ_label = np.chararray((Qa_maxJ.shape[0], Qa_maxJ.shape[1]), unicode=True)
        for x in np.arange(0, Qa_maxJ.shape[0]):
            for y in np.arange(0, Qa_maxJ.shape[1]):
                # for invalid values label as "-"
                if ((j==1) & (x<=7)):
                    Qa_maxJ_label[x,y] = "-"                                        
                else:
                    Qa_maxJ_label[x,y] = action_dict(Qa_maxJ[x,y], short=True)

        # obtain the relevant Q values - hiding non-used states (4- & 22+ for player / 0 for dealer)
        g = sns.heatmap(Qa_maxJ, cmap='Reds', annot=Qa_maxJ_label, fmt ='',
                        cbar=False, ax=axes[j], vmin=0, vmax=5)
        g.set_title("{}{}".format(j_labels[j], add_title))
        g.set_xlabel('Dealer initial card')
        # set agent hand value range of 4-21 inclusive, dealer 2-11 inclusive
        g.set_yticklabels(np.arange(4,22), rotation=0)
        g.set_xticklabels(np.arange(1,11), rotation=0)  # where Ace = 1 for now
        # only show y label on first plot
        if j == 0:
            g.set_ylabel('Player hand value')
    plt.show()

    # legend to remind what the possible actions are
    print("Legend:\nS = Stick || H = Hit || D = Double  || Initial card 1 = Ace")
    
make_adviceChart(Q_in = None, add_title='\nHuman strategy', smeChart=True)<jupyter_output><empty_output><jupyter_text>Notice how the dealer's initial card has a large impact on the strategy our agent will follow, sticking a whole 5 points earlier for a 6 versus a 7 for example. Also note how much more aggressive the human player will be once it has a useable ace, choosing to double down over many more hand values.Now that we have visualised the human strategy, we will form a 3D matrix aligned to the format of the Q-matrices that our agents were using. Rather than storing estimated values, however, it will simply store the ID of the action to take for any given state. This will help us insert the static strategy into the environment with minimal code changes required.<jupyter_code># load files and stack into a 3D array
Qh_no_ace = np.array(pd.read_csv('additional_analysis/sme_advice_noUseableAce.csv'))
Qh_ace = np.array(pd.read_csv('additional_analysis/sme_advice_UseableAce.csv'))
Qh = np.dstack([Qh_no_ace, Qh_ace])<jupyter_output><empty_output><jupyter_text>Now we can simulate a large number of games and calculate the average reward and success rate for this human strategy. Since the strategy is fixed and no learning is occurring, we do not need to play the full number of games that the agents played. We must however ensure the strategy is evaluated over the same number of simulations: 25,000.To get a better estimate of these evaluation metrics, we will perform the each round of games 400 times and take the average.<jupyter_code>num_simulation_rounds = 400
list_meanR = []
list_success = []
for t in np.arange(0, num_simulation_rounds):
    #create list to contain total rewards and steps per episode
    rList = []
    for i in range(EVAL_LAST_N_EPS):
        #Reset environment and get first new observation
        s = env.reset()
        rAll = 0
        d = False
        j = 0

        # game will not exceed 20 moves, limit while loop out of caution
        while j < 20:
            j+=1

            # choose action per fixed strategy
            # need to adjust the indices slightly for true alignment
            a = Qh[s[0], s[1]-1, s[2]]

            #Get new state and reward from environment
            player_hand_old = env.player.copy()
            s1,r,d,_ = env.step(a)

            # update rewards and state
            rAll += r
            s = s1

            # end game when a winner/loser found
            if d == True:            
                # For each of the final 5 episodes, show the game for inspection
                if ((i >= (NUM_EPISODES_TRAIN-5)) & (t==0)):
                    show_ep(env.player, env.dealer, r, i)
                break

        # append reward to list
        rList.append(rAll)

    # show overall success history of fixed strategy for first simulation round
    if t == 0:
        plt.figure(figsize=(12,4))
        plt.plot(np.cumsum(rList))
        plt.title('Performance vs. episodes (Human strategy - first simulation round)')
        plt.ylabel('Total Returns')
        plt.xlabel('Episode number')
        plt.show()

    meanR = meanR_lastN(rList, EVAL_LAST_N_EPS)
    ts_success, mean_success_rate = calc_success_rate2(rList, produce_ts=False)
    
    if t % 25 == 0:
        print("Round %i. Mean rewards of %.5f over %i games --> success rate: %.5f" % (t, meanR,
                                                                          EVAL_LAST_N_EPS,
                                                                          mean_success_rate))
    # add new metrics to running lists
    list_meanR.append(meanR)
    list_success.append(mean_success_rate)
def summarise_eval(meanR_list, success_list, label="HUMAN"):
    # now show averages of evaluation metrics after all the simulations
    mean_meanR = np.array(meanR_list).mean()
    mean_success_rate =  np.array(success_list).mean()
    mean_expected_value = expected_value_funds(mean_meanR)
    print("FINAL EVALUATION METRICS FOR {} STRATEGY:".format(label))
    print("Mean of mean rewards: %.5f" % mean_meanR)
    print("Expected value of £100 after 100 games: £%.2f" % mean_expected_value)
    print("Mean of success rates: %.5f" % mean_success_rate)
    return mean_meanR, mean_success_rate, mean_expected_value

# show evaluation summary
mean_meanR, mean_success_rate, mean_expected_value = summarise_eval(list_meanR, list_success)

# plot distribution of metrics per round of simulations
f, (ax1, ax2) = plt.subplots(1,2, sharey=True)
ax1.hist(np.array(list_meanR))
ax1.axvline(x=mean_meanR, color="r")
ax2.hist(np.array(list_success))
ax2.axvline(x=mean_success_rate, color="r")
ax1.set_ylabel("freq")
ax1.set_xlabel("mean rewards")
ax2.set_xlabel("success rate")
plt.show()<jupyter_output>FINAL EVALUATION METRICS FOR HUMAN STRATEGY:
Mean of mean rewards: -0.03349
Expected value of £100 after 100 games: £96.65
Mean of success rates: 0.48357<jupyter_text>So there we have it - the human strategy consistently beats the strategies our agents have learnt so far.But let's try one more thing before we conclude that the human strategy is definitely the optimal strategy to use. Let's try training a new agent over 1 million games (vs. 125,000 before), using the parameters we identified as best from grid search, but this time also adding a very slow decay into the learning rate at the end of each game after 250,000 games. This should allow the agent to start fine-tuning its approach once it has accumulated a strong foundation, rather than cycling between different strategies each time it loses a round due to "bad luck".You'll notice I already included the below code in the playGame() function, which will implement the desired decrease in learning rate now that we are playing more than 250,000 games:```if d == True:       ...    ...    learn_decay_after_ep = 250000    if (i >= learn_decay_after_ep):        lr *= 0.99999```Warning: The next block could take up to an hour to complete and may be difficult to interrupt!<jupyter_code>### Set scope of training and evaluation
# number of episodes to train agent over
NUM_EPISODES_TRAIN = 1000000           

# evaluate agent with mean rewards over last (N) episodes
EVAL_LAST_N_EPS = 25000                 

# Raise verbosity back up
VERBOSE =2

# Set hyper-parameters: grid format will be beneficial when we optimize these 
grid = ParameterGrid({"dbl_Q" : [1],                # 1 to use Double Q learning
                          "lr": [0.01],             # learning rate
                          "y": [0.25],              # gamma
                          "policy": "b",            # "e" for e-greedy, "b" for boltzmann
                          "policy_param":  [1],     # epsilon / tau (inverse temperature)
                          "num_episodes": [NUM_EPISODES_TRAIN],
                          "cb_over_num_eps": [EVAL_LAST_N_EPS],
                          "verbose": [VERBOSE]})

# just one set of parameters to go through at this stage
grid_size = len(grid)
all_eval_metrics, fine_tuned_agent_filepath, fine_tuned_agent_filepath_cb = train_agent(0, grid[0], True)<jupyter_output>---------------------------
Now training agent 1/1
DblQ: 1.00 | LR: 0.010 | g: 0.250 | pol: b | pol_param: 1.00000
Ep: 0	Mean reward over last 25000 episodes: 1.0. (b-policy @ 1.00001)
Ep: 200000	Mean reward over last 25000 episodes: -0.11824. (b-policy @ 5.84809)
Ep: 400000	Mean reward over last 25000 episodes: -0.05332. (b-policy @ 34.19982)
Ep: 600000	Mean reward over last 25000 episodes: -0.03764. (b-policy @ 200.0)
Ep: 800000	Mean reward over last 25000 episodes: -0.05456. (b-policy @ 200.0)
--- SUMMARY OF EP #999995 ---
Player:	[ 6  8 18] ---
Dealer:	[ 4 14 18] ---
[6, 2, 10]
[4, 10, 4]
--- SUMMARY OF EP #999996 ---
Player:	[ 3  4 14 20] !!!
Dealer:	[10 15 24] XXX
[3, 1, 10, 6]
[10, 5, 9]
--- SUMMARY OF EP #999997 ---
Player:	[10 20] !!!
Dealer:	[ 3 14 17] 
[10, 10]
[3, 11, 3]
--- SUMMARY OF EP #999998 ---
Player:	[10 13 22] XXX
Dealer:	[ 3 13] !!!
[10, 3, 9]
[3, 10]
--- SUMMARY OF EP #999999 ---
Player:	[ 3  5  8 18] !!!
Dealer:	[ 4 12 22] XXX
[3, 2, 3, 10]
[4, 8, 10]<jupyter_text>Let's do the same number of simulations with this new agent, where learning is turned off so that we get a clear understanding of its strategy's long-run performance.<jupyter_code># load the agent's exported Q-matrix back into memory
pickle_in = open(fine_tuned_agent_filepath,"rb")
best_Q = pickle.load(pickle_in)

# same for the callback version
pickle_in = open(fine_tuned_agent_filepath_cb,"rb")
best_Q_cb = pickle.load(pickle_in)
list_meanR_agent = []
list_success_agent = []
for t in np.arange(0, num_simulation_rounds):
    #create list to contain total rewards and steps per episode
    rList = []
    for i in range(EVAL_LAST_N_EPS):
        #Reset environment and get first new observation
        s = env.reset()
        rAll = 0
        d = False
        j = 0

        # game will not exceed 20 moves, limit while loop out of caution
        while j < 20:
            j+=1

            # choose action per fixed strategy
            a = np.argmax(best_Q[s[0], s[1], s[2]])

            #Get new state and reward from environment
            player_hand_old = env.player.copy()
            s1,r,d,_ = env.step(a)

            # update rewards and state
            rAll += r
            s = s1

            # end game when a winner/loser found
            if d == True:            
                # For each of the final 5 episodes, show the game for inspection
                if ((i >= (NUM_EPISODES_TRAIN-5)) & (t==0)):
                    show_ep(env.player, env.dealer, r, i)
                break

        # append reward to list
        rList.append(rAll)

    # show overall success history of fixed strategy for first simulation round
    if t == 0:
        plt.figure(figsize=(12,4))
        plt.plot(np.cumsum(rList))
        plt.title('Performance vs. episodes (AI strategy - first simulation round)')
        plt.ylabel('Total Returns')
        plt.xlabel('Episode number')
        plt.show()

    meanR = meanR_lastN(rList, EVAL_LAST_N_EPS)
    ts_success, mean_success_rate = calc_success_rate2(rList, produce_ts=False)
    
    if t % 25 == 0:
        print("Round %i. Mean rewards of %.5f over %i games --> success rate: %.5f" % (t, meanR,
                                                                          EVAL_LAST_N_EPS,
                                                                          mean_success_rate))
    # add new metrics to running lists
    list_meanR_agent.append(meanR)
    list_success_agent.append(mean_success_rate)
# now show averages of evaluation metrics after all the simulations
mean_meanR_agent, mean_success_rate_agent, mean_expected_value_agent = summarise_eval(list_meanR_agent,
                                                                                      list_success_agent,
                                                                                      label="AI")

## Show the human / AI distributions side-by-side for comparison
# form consistent bins for both distributions
num_bins = 15
low = np.min([np.min(list_meanR), np.min(list_meanR_agent)])
high = np.max([np.max(list_meanR), np.max(list_meanR_agent), 0.0])
bins = np.linspace(low, high, num_bins)
low = np.min([np.min(list_success), np.min(list_success_agent)])
high = np.max([np.max(list_success), np.max(list_success_agent), 0.5])
bins2 = np.linspace(low, high, num_bins)

# plot as histograms
f, (ax1, ax2) = plt.subplots(1,2, sharey=True)
ax1.hist(np.array(list_meanR), bins=bins, alpha=0.25, label="Human")
ax1.hist(np.array(list_meanR_agent), bins=bins,  alpha=0.25, label="AI")
ax1.axvline(x=mean_meanR, color="purple", linestyle="--", label="Human mean")
ax1.axvline(x=mean_meanR_agent, color="red", linestyle="--", label="AI mean")
ax2.hist(np.array(list_success), bins=bins2, alpha=0.25, label="Human")
ax2.hist(np.array(list_success_agent), bins=bins2, alpha=0.25, label="AI")
ax2.axvline(x=mean_success_rate, color="purple", linestyle="--", label="Human mean")
ax2.axvline(x=mean_success_rate_agent, color="red", linestyle="--", label="AI mean")
ax1.set_ylabel("freq")
ax1.set_xlabel("mean rewards")
ax2.set_xlabel("success rate")
ax2.legend(loc='center left', bbox_to_anchor=(1.1, 0.5))
plt.show()

# summarise agent performance as pc expected value funds vs human expected value funds
agent_pc_expected_value = mean_expected_value_agent / mean_expected_value
print("Mean AI performance relative to human:\t%.3f pc" % (100*agent_pc_expected_value))<jupyter_output>FINAL EVALUATION METRICS FOR AI STRATEGY:
Mean of mean rewards: -0.03613
Expected value of £100 after 100 games: £96.39
Mean of success rates: 0.48169<jupyter_text>While the extra training at lower learning rate certainly helped our agent, it is still not quite as good as the human strategy. If we take the average "expected value of £100 after 100 games" for the human and treat it as the benchmark for perfect performance (~ "100% accuracy"), this allows us to calculate a neat "accuracy" metric for the agent. As shown above, **the agent has learnt a strategy which is 99.7% accurate**.  7. DiscussionWe may not have managed to create AI which can outperform humans at Blackjack, but we have shown how Q-learning and Double Q-learning under two different exploration policies can lead to outcomes which are nearly indistinguishable. One important question we've not yet looked at is how do the strategies of a human and our best agent actually differ? Let's re-plot the advice chart we made earlier for the human strategy, then use the same function to plot the equivalent for the best agent's strategy.We see that there are many similarities overall, with the dealer's initial card having a big impact on both strategies. The biggest difference appears to concern Doubling down while holding a usable Ace, where the neat region in which a human would do this is broken up for the best agent. The Stick at 18, 4 surrounded by Doubles is particularly telling, and suggests that some cycling between actions has occurred after "unlucky" hands saw outcomes the agent didn't expect. A slightly different set of learning parameters may avoid this.<jupyter_code>make_adviceChart(Q_in = None, add_title='\nHuman strategy', smeChart=True)
make_adviceChart(Q_in = best_Q, add_title='\nBest agent strategy', smeChart=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09737017895440003, 'fraction_numerical': 0.025595049748620046, 'mean_word_length': 2.5532167395377887, 'pattern_counts': {'":': 34, '<': 76, '<?xml version=': 0, '>': 99, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 52, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '570459', 'license': 'MIT', 'n_tokens_mistral': 17960, 'n_tokens_neox': 16556, 'n_words': 6123, 'path': 'learn_blackjack.ipynb', 'repo_name': 'slmwest/blackjack', 'size': '56905'}