<jupyter_start><jupyter_text>Bandit ProblemBackgroundThe Bandit problem is to illustrate a scenario whereby you have to choose a decision which will result in getting you the maximum rewards. The decision may be wrong or correct which would not be known until you take it, however when you have to re-iterate taking the same decisions over and over again. You evdntually get an intuition (or a quantitative definite answer) to which decision could give you a better result and you learn over-time.Bandit here refers to the slot machines in Casinos. The slot machine has an arm and a set of reels with images on it. When an arm is pulled, by a game a chance a combination of images are selected in the reels. These combination of images decide the reward given to the player. This machine is also called a one-armed bandit, because of the tendency of a player to lose large amount of money.Problem Description  Implementation of Epsilon greedy strategy<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
class Bandit:
  def __init__(self,trueMean):
    self.trueMean = trueMean
    self.mean = 0
    self.N = 0
    self.pull_values = []

  def pull(self):
    reward = np.random.randn() + self.trueMean
    return reward
  
  def update(self,x):
    self.N += 1;
    self.mean = (1-(1/self.N))*self.mean + 1/self.N*x<jupyter_output><empty_output><jupyter_text>The class Bandit has a trueMean which we intialise during program run. **Question:****Why do we have a gussian normal distribution added to the trueMean?**It is the reward function. So, I think the goal here is to get a large number as a reward.*(I will fill this in has and when I understand more about this...)*Next is the update function, which updates our mean of the machine. It basically finds mean for each step and adds the mean upto N-1 steps to the current N to give the new mean. This avoids keeping track of all Xi for N values.<jupyter_code>def run_game(m1,m2,m3, eps, N):
  bandits = [Bandit(m1),Bandit(m2),Bandit(m3)]
  data = np.empty(N)
  chosen_bandit = np.empty(3)
  for i in range(N):
    p = np.random.random();
    if p < eps:
      # explore: choose randomly between the three slot machines
      j = np.random.choice(3)
    else:
      # exploit: choose the best slot machine that gives a good reward
      j = np.argmax([b.mean for b in bandits])
    # now that we have selected the slot machine, lets pull its arm
    # to get a reward
    x = bandits[j].pull()
    bandits[j].update(x)
    # for plotting the reward
    data[i] = x
    # to check which slot machine was pulled most, store the frequency of each bandit
    chosen_bandit[j] = chosen_bandit[j]+1

  print("Sample reward values: ")
  print(data[100:150])
  cumulative_average = np.cumsum(data)/(np.arange(N) + 1)

  plt.plot(data)
  plt.xscale('log')
  plt.title("Rewards on each pull (eps ="+str(eps)+")")
  plt.show()

  plt.bar(["Bandit 1", "Bandit 2", "Bandit 3"], chosen_bandit)
  plt.title("Frequency of bandit chosen (eps = "+str(eps)+")")
  plt.show()

  print("plotting cum avg on log scale (eps= "+str(eps)+")...")
  plt.plot(cumulative_average, label="cumulative average")
  plt.plot(np.ones(N)*m1, label="m1")
  plt.plot(np.ones(N)*m2, label="m2")
  plt.plot(np.ones(N)*m3, label="m3")
  plt.xscale('log')
  plt.legend()
  plt.show()

  
  # this mean is just to observe mean of the bandit converging to the true mean specified in the args
  print("Mean of bandits: \n")
  for b in bandits:
    print(b.mean)

  return cumulative_average

if __name__ == '__main__':
  c_1 = run_game(1.0,2.0,3.0, 0.1, 1000000)
  c_5 = run_game(1.0,2.0,3.0, 0.5, 1000000)
  c_8 = run_game(1.0,2.0,3.0, 0.8, 1000000)

  plt.plot(c_1, label='eps = 0.1')
  plt.plot(c_5, label='eps = 0.5')
  plt.plot(c_8, label='eps = 0.8')
  plt.legend()
  plt.title("Plot of reward over pulls on a Log scale")
  plt.xlabel("No of pulls")
  plt.ylabel("Reward")
  plt.xscale('log')
  plt.show()<jupyter_output>Sample reward values: 
[1.69181099 5.15992275 2.26256194 2.98857767 2.30411076 3.66620785
 0.24560123 2.52775795 2.98855246 2.74988631 4.20774152 3.09217383
 4.04052535 1.34038739 2.94647288 3.60329228 2.03666363 6.14305359
 3.1103554  2.92797574 3.34240003 4.0449775  1.05616961 3.81835097
 3.23470795 4.78241578 2.09273756 3.9518727  1.86433913 2.1371099
 1.69477703 3.32666091 3.5343779  1.50488357 3.86618001 1.54526284
 2.60540257 2.88216253 3.21702527 4.17542792 2.82758293 3.05386842
 3.36223667 2.91163678 3.39514273 3.98625228 2.10137433 1.96132006
 5.09486299 4.00166017]<jupyter_text>When the eps value is high, we tend to expore more and exploit less. The inverse is true for low epsilon value.At high eps value, half the time we have chosen the less optimal bandits while  at eps=0.1 we are exploiting i.e. choosing the bandit which has a large mean among the three. **Question****I am very confused over the true-mean intialised for each bandit. The reward function arbitarily adds a guassian number to the mean. So in case of the m1,m2,m3 of 1,2,3, when the exploit is done more (eps=0.1) for 90% of the cases, the bandit with the larger mean would be chosen which would be the third one which has large true-mean. Then, we end up choosing the third bandit more as shown in figure above. Whereas, if it was high epsilon(eps=0.8) then for 80% of the cases we explore more, resulting in lesser optimal rewards and pulling other two bandits more. So, here true-mean actually means reward-weight?**(This is what I think happens) The actual experiment programatically should be, by epsilon strategy choose one of the bandits resolving the explore and exploit dilemma. Now, when an arm is pulled the bandit should pull give a reward. The reward is randomly generated and as we do more iterations we get an intuition of which bandit has a higher chance of giving a reward. Therefore, here we are already riding on the assumption that the bandits are biased and that some give better result than others. This idea is what is captured in the true mean. The true-mean adds a bias to the bandits. The idea of the above experiment is to find the bandit with this higher bias. Now, for finding it we can do two approaches1. Run bandits for a sizable sample N of statistical significance and get the mean of rewards in the end after N runs. 2. Run for N times and converge towards finding the bandit with the higher bias. To converge we have to either explore or exploit. Which is to say, we have to pull one of the possible less-optimal bandit once in a while so that we can see if it gives a high reward else we go with the bandit which according to the currently collected mean of each machine has highest value. After a certain size of samples of statistical significance, we can do just exploit and don't have to explore anymore.  Implementation of Epsilon Optimistic Strategy<jupyter_code>class Bandit_Optimistic:
  def __init__(self,true_mean, upper_limit):
    self.true_mean = true_mean
    self.mean = upper_limit
    self.N = 1

  def pull(self):
    reward = np.random.randn() + self.true_mean
    return reward
  
  def update(self,x):
    self.N += 1;
    self.mean = (1-(1/self.N))*self.mean + 1/self.N*x
  

def run_game_optimistic(m1,m2,m3, N):
  bandits = [Bandit_Optimistic(m1,10),Bandit_Optimistic(m2,10),Bandit_Optimistic(m3,10)]
  data = np.empty(N)
  chosen_bandit = np.empty(3)
  for i in range(N):
    j = np.argmax([b.mean for b in bandits])
    # now that we have selected the slot machine, lets pull its arm
    # to get a reward
    x = bandits[j].pull()
    bandits[j].update(x)
    # for plotting the reward
    data[i] = x
    # to check which slot machine was pulled most, store the frequency of each bandit
    chosen_bandit[j] = chosen_bandit[j]+1

  print("Sample reward values: ")
  print(data[100:150])
  cumulative_average = np.cumsum(data)/(np.arange(N) + 1)

  plt.plot(data)
  plt.xscale('log')
  plt.title("Rewards on each pull")
  plt.show()

  plt.bar(["Bandit 1", "Bandit 2", "Bandit 3"], chosen_bandit)
  plt.title("Frequency of bandit chosen")
  plt.show()

  print("plotting cum avg on log scale...")
  plt.plot(cumulative_average, label="cumulative average")
  plt.plot(np.ones(N)*m1, label="m1")
  plt.plot(np.ones(N)*m2, label="m2")
  plt.plot(np.ones(N)*m3, label="m3")
  plt.xscale('log')
  plt.legend()
  plt.show()

  
  # this mean is just to observe mean of the bandit converging to the true mean specified in the args
  print("Mean of bandits: \n")
  for b in bandits:
    print(b.mean)

  return cumulative_average

if __name__ == '__main__':
  c_1 = run_game_optimistic(1.0,2.0,3.0, 1000000)
  c_5 = run_game_optimistic(1.0,2.0,3.0, 1000000)
  c_8 = run_game_optimistic(1.0,2.0,3.0, 1000000)

  plt.plot(c_1, label='eps = 0.1')
  plt.plot(c_5, label='eps = 0.5')
  plt.plot(c_8, label='eps = 0.8')
  plt.legend()
  plt.title("Plot of reward over pulls on a Log scale")
  plt.xlabel("No of pulls")
  plt.ylabel("Reward")
  plt.xscale('log')
  plt.show()<jupyter_output>Sample reward values: 
[3.56718093 2.85504895 1.72459733 3.62122484 1.71279501 4.12661411
 3.29895472 2.44740203 1.83554462 2.57233507 3.44973743 3.35581175
 5.43799463 1.779712   3.49490887 4.26607673 1.81060926 1.61725881
 1.74771605 3.13444712 3.9789087  2.67517409 3.14707135 2.40504222
 2.84017169 3.42804353 2.33613733 3.5495276  2.78423749 2.63487094
 1.57168196 3.10275829 1.2053146  2.78031066 3.64967084 4.02838711
 2.62328907 3.61434982 2.08538773 1.94908383 1.8685853  3.7574228
 2.99104815 3.56318857 3.58001496 2.55077133 2.99191842 4.25189555
 2.83069897 3.73831468]
{'fraction_non_alphanumeric': 0.09823203263939742, 'fraction_numerical': 0.11308714300659065, 'mean_word_length': 3.8676171079429733, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '399861', 'license': 'Unlicense', 'n_tokens_mistral': 3971, 'n_tokens_neox': 3266, 'n_words': 1200, 'path': 'Three_armed_bandit_problem.ipynb', 'repo_name': 'vishnudas-raveendran/AI_RL_Python', 'size': '9559'}