<jupyter_start><jupyter_text>Multi-armed bandits: A gentle introduction to reinforcement learning<NAME> Data Science7 March 2019<EMAIL> This notebook demonstrates a couple of reinforcement learning policies for the multi-armed bandit problem. It is intended to be used as part of a presentation on this topic.  Imports<jupyter_code>%load_ext autoreload
%autoreload 2
import os
from bandits import Bandits, RandomStrategy, epsilonGreedyStrategy, BayesianStrategy
from pymc import rbeta
from scipy.stats.distributions import beta
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
mpl.rcParams['figure.dpi'] = 300
from tqdm import tqdm<jupyter_output><empty_output><jupyter_text>Demo of Beta distribution First, let's take a look at the uniform prior for the Beta distribution.<jupyter_code>fig, ax = plt.subplots(figsize=(3,2))

xpoints = np.linspace(0.0,1.0,100)
ax.plot(
    xpoints,
    beta.pdf(x=xpoints,a=1,b=1),
)
ax.set_xlabel('bandit arm parameter\n(probability of distributing reward)')
ax.set_ylabel('uniformed prior\nprobability density')
ax.set_xlim([0,1])
ylim = ax.get_ylim()
ax.set_ylim([0,ylim[1]])<jupyter_output><empty_output><jupyter_text>Next I'll plot some Beta distributions with different parameters.<jupyter_code>fig, ax = plt.subplots(figsize=(5,3))

xpoints = np.linspace(0.0,1.0,100)
ax.plot(
    xpoints,
    beta.pdf(x=xpoints,a=2,b=8),
    label='Beta(2,8)',
    color='blue',
)
ax.plot(
    xpoints,
    beta.pdf(x=xpoints,a=8,b=2),
    label='Beta(8,2)',
    color='green',
)
ax.plot(
    xpoints,
    beta.pdf(x=xpoints,a=5,b=5),
    label='Beta(5,5)',
    color='magenta',
)
ax.set_xlabel('bandit arm parameter\n(probability of distributing reward)')
ax.set_ylabel('probability density')
ax.set_xlim([0,1])
ylim = ax.get_ylim()
ax.set_ylim([0,ylim[1]])
ax.legend()<jupyter_output><empty_output><jupyter_text>Now simulate some results from a bandit. We'll say this bandit has a success probability of 0.6. Run the cell below to reset the bandit arm's Beta distribution to an uniformed prior.<jupyter_code>bandit_arm_param = 0.6

num_successes = 0
num_failures = 0<jupyter_output><empty_output><jupyter_text>Run this cell repeatedly to see how the Beta distribution evolves as we get successes and failures when pulling the bandit arm.<jupyter_code>success = np.random.binomial(1,bandit_arm_param)

if success:
    num_successes += 1
else:
    num_failures += 1
    
fig, ax = plt.subplots(figsize=(3,2))

xpoints = np.linspace(0.0,1.0,100)
ax.plot(
    xpoints,
    beta.pdf(x=xpoints,a=1+num_successes,b=1+num_failures),
)
ax.set_xlabel('bandit arm parameter\n(probability of distributing reward)')
ax.set_ylabel('probability density\n(after {} successes and {} failures)'.format(num_successes, num_failures))
ax.set_xlim([0,1])
ylim = ax.get_ylim()
ax.set_ylim([0,ylim[1]])<jupyter_output><empty_output><jupyter_text>Compare performance of policies by running some experiments  Set up experiment parameters<jupyter_code>NUM_EXPERIMENTS = 50
NUM_PULLS = 10000
BANDITS_PROBS = [0.4, 0.5, 0.5, 0.5, 0.5, 0.6]
EPS_1 = 0.1
EPS_2 = 0.05<jupyter_output><empty_output><jupyter_text>Build some bandits<jupyter_code>bandits = Bandits(BANDITS_PROBS)
len(bandits)
bandits.p<jupyter_output><empty_output><jupyter_text>Run the experiment<jupyter_code># I'll keep a cumulative sum in these, then divide by the number of experiments at the end
random_total_regret = np.zeros(NUM_PULLS)
eps1_greedy_total_regret = np.zeros(NUM_PULLS)
eps2_greedy_total_regret = np.zeros(NUM_PULLS)
bayes_total_regret = np.zeros(NUM_PULLS)

random_total_reward = np.zeros(NUM_PULLS)
eps1_greedy_total_reward = np.zeros(NUM_PULLS)
eps2_greedy_total_reward = np.zeros(NUM_PULLS)
bayes_total_reward = np.zeros(NUM_PULLS)

for experiment in tqdm(range(NUM_EXPERIMENTS)):
    random_policy = RandomStrategy(bandits)
    random_policy.sample(NUM_PULLS)
    random_total_regret += np.cumsum(random_policy.regret)
    random_total_reward += np.cumsum(random_policy.score)
    
    eps1_greedy_policy = epsilonGreedyStrategy(bandits, EPS_1)
    eps1_greedy_policy.sample(NUM_PULLS)
    eps1_greedy_total_regret += np.cumsum(eps1_greedy_policy.regret)
    eps1_greedy_total_reward += np.cumsum(eps1_greedy_policy.score)
    
    eps2_greedy_policy = epsilonGreedyStrategy(bandits, EPS_2)
    eps2_greedy_policy.sample(NUM_PULLS)
    eps2_greedy_total_regret += np.cumsum(eps2_greedy_policy.regret)
    eps2_greedy_total_reward += np.cumsum(eps2_greedy_policy.score)
    
    bayes_policy = BayesianStrategy(bandits)
    bayes_policy.sample(NUM_PULLS)
    bayes_total_regret += np.cumsum(bayes_policy.regret)
    bayes_total_reward += np.cumsum(bayes_policy.score)
    
random_total_regret = random_total_regret/NUM_EXPERIMENTS
eps1_greedy_total_regret = eps1_greedy_total_regret/NUM_EXPERIMENTS
eps2_greedy_total_regret = eps2_greedy_total_regret/NUM_EXPERIMENTS
bayes_total_regret = bayes_total_regret/NUM_EXPERIMENTS

random_total_reward = random_total_reward/NUM_EXPERIMENTS
eps1_greedy_total_reward = eps1_greedy_total_reward/NUM_EXPERIMENTS
eps2_greedy_total_reward = eps2_greedy_total_reward/NUM_EXPERIMENTS
bayes_total_reward = bayes_total_reward/NUM_EXPERIMENTS<jupyter_output>100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [03:09<00:00,  3.78s/it]<jupyter_text>Review the results<jupyter_code>fig, ax = plt.subplots(figsize=(5,3))

# ax.plot(random_total_regret,
#               color='gray',
#               label='random',
#              )

ax.plot(eps1_greedy_total_regret,
              color='magenta',
              label='$\epsilon$-greedy ($\epsilon=${})'.format(EPS_1),
             )

ax.plot(eps2_greedy_total_regret,
              color='blue',
              label='$\epsilon$-greedy ($\epsilon=${})'.format(EPS_2),
             )

ax.plot(bayes_total_regret,
              color='green',
              label='Bayesian Bandit',
             )

for direction in ['left',"right", "top", 'bottom']:
    # hides borders
    ax.spines[direction].set_visible(False)

plt.ylabel('Expected total regret after $n$ pulls')
plt.xlabel('Number of pulls ($n$)')

plt.legend(loc='best', framealpha=1, frameon=True)

plt.tight_layout()

fig.savefig(os.path.join('output','plots','expected_total_regret.png'))
fig, ax = plt.subplots(figsize=(5,3))

ax.plot(random_total_reward,
              color='gray',
              label='random',
             )

ax.plot(eps1_greedy_total_reward,
              color='magenta',
              label='$\epsilon$-greedy ($\epsilon=${})'.format(EPS_1),
             )

ax.plot(eps2_greedy_total_reward,
              color='blue',
              label='$\epsilon$-greedy ($\epsilon=${})'.format(EPS_2),
             )

ax.plot(bayes_total_reward,
              color='green',
              label='Bayesian Bandit',
             )

for direction in ['left',"right", "top", 'bottom']:
    # hides borders
    ax.spines[direction].set_visible(False)

plt.ylabel('Expected total reward after $n$ pulls')
plt.xlabel('Number of pulls ($n$)')

plt.legend(loc='best', framealpha=1, frameon=True)

plt.tight_layout()

fig.savefig(os.path.join('output','plots','expected_total_rewards.png'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13107064017660044, 'fraction_numerical': 0.021661147902869757, 'mean_word_length': 4.966255144032922, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '261438', 'license': 'MIT', 'n_tokens_mistral': 2894, 'n_tokens_neox': 2702, 'n_words': 448, 'path': 'MAB_RL_demo_presentation.ipynb', 'repo_name': 'mbloem/multi-armed-bandits', 'size': '7282'}