<jupyter_start><jupyter_text>DAT257x: Reinforcement Learning Explained Lab 2: Bandits Exercise 2.2B: Optimistic Greedy<jupyter_code>import numpy as np
import sys

if "../" not in sys.path:
    sys.path.append("../") 

from lib.envs.bandit import BanditEnv
from lib.simulation import Experiment
#Policy interface
class Policy:
    #num_actions: (int) Number of arms [indexed by 0 ... num_actions-1]
    def __init__(self, num_actions):
        self.num_actions = num_actions
    
    def act(self):
        pass
        
    def feedback(self, action, reward):
        pass
#Greedy policy
class Greedy(Policy):
    def __init__(self, num_actions):
        Policy.__init__(self, num_actions)
        self.name = "Greedy"
        self.total_rewards = np.zeros(num_actions, dtype = np.longdouble)
        self.total_counts = np.zeros(num_actions, dtype = np.longdouble)
    
    def act(self):
        current_averages = np.divide(self.total_rewards, self.total_counts, where = self.total_counts > 0)
        current_averages[self.total_counts <= 0] = 0.5      #Correctly handles Bernoulli rewards; over-estimates otherwise
        current_action = np.argmax(current_averages)
        return current_action
        
    def feedback(self, action, reward):
        self.total_rewards[action] += reward
        self.total_counts[action] += 1<jupyter_output><empty_output><jupyter_text>Now let's implement an optimistic greedy policy based on the policy interface. The optimistic greedy policy initialize the ra to a large initial value R, which is implemented in the __init__() function, and then play the greedy algorithm. We have given you some boiler plate code, you only need to modify the part as indicated.<jupyter_code>#Optimistic Greedy policy
class OptimisticGreedy(Greedy):
    def __init__(self, num_actions, initial_value):
        Greedy.__init__(self, num_actions)
        self.name = "Optimistic Greedy"
        self.initial_value = initial_value
        self.total_rewards[self.total_counts <= 0] = self.initial_value
        
    def act(self):
        current_averages = np.divide(self.total_rewards, self.total_counts, where = self.total_counts > 0)
        current_averages[self.total_counts <= 0] = self.initial_value
        current_action = np.argmax(current_averages)
        return current_action<jupyter_output><empty_output><jupyter_text>Let's prepare the simulation. We'll use the same parameters as the exercise with the epsilon greedy.<jupyter_code>evaluation_seed = 5016
num_actions = 10
trials = 10000
distribution = "bernoulli"<jupyter_output><empty_output><jupyter_text>First, let's use R = 0. Run the simulation and observe the results.<jupyter_code>R = 1
env = BanditEnv(num_actions, distribution, evaluation_seed)
agent = OptimisticGreedy(num_actions, R)
experiment = Experiment(env, agent)
experiment.run_bandit(trials)<jupyter_output>Distribution: bernoulli [0.5061565  0.74836123 0.53065236 0.37446716 0.88168477 0.83849367
 0.3951277  0.13217982 0.44509856 0.03459039]
Optimal arm: 4<jupyter_text>What about if R is a very large number? Say 10000, the same number as the number of trials? Run the simulation again and observe the results. Now, try several different number of R (1, 3, 5). Run the simulations and observe the results. Which epsilon performs best with this problem? Now let's prepare another simulation by setting a different seed.<jupyter_code>evaluation_seed = 1239
num_actions = 10
trials = 10000
distribution = "bernoulli"<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08300057045065602, 'fraction_numerical': 0.03907586993725043, 'mean_word_length': 4.142228739002933, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '27516', 'license': 'Unlicense', 'n_tokens_mistral': 1201, 'n_tokens_neox': 1074, 'n_words': 316, 'path': 'Module 2/Ex2.2B Optimistic Greedy.ipynb', 'repo_name': 'tycho01/Reinforcement-Learning-Explained', 'size': '3506'}