<jupyter_start><jupyter_text>**Bernoulli Bandits Class** The [Bernoulli Bandits](https://en.wikipedia.org/wiki/Multi-armed_banditVariations) Class defined below allows users to experiment with applying [Thompson Sampling](https://en.wikipedia.org/wiki/Thompson_sampling) to the classical multi-armed bandits problem of reinforcement learning (explore vs exploit).It allows the user to select the number of bandits the agent can choose from and a range for the probabilities of winning for each machine to be selected from randomly.This helps build an intuition about the limits of Thompson Sampling, thus helping practitoners to apply the technique more effectively.  For instance, in website optimization it is beneficial to know how many variations that are how close in likelihood to convert can be tested effectively at once, and how well Thompson Sampling will do at finding the best choice and/or limiting regret.It is the authors belief that web optimization is a truly online problem and that Thompson Sampling is superior to A/B testing for this purpose (though less used).  Also, that reinforcement learning, broadly, will surpass traditional methods and that Thompson Sampling is a perfect place to start for beginners.  **Python Implementation**<jupyter_code>import numpy as np
import random
import pandas as pd
import matplotlib.pyplot as plt

class N_Bernoulli_Bandit:
    def __init__(self, total_reward = 0, rounds = 0, num_machines = 3, machine_played = [], wins = [], losses = [], machine_probs = []):

        self.total_reward = total_reward
        self.machine_played = machine_played
        self.wins = wins
        self.losses = losses
        self.machine_probs = machine_probs
        self.rounds = rounds
        self.num_machines = num_machines

    def simulate(self, num_machines = 3, rounds = 1000, prob_range = [0,1]):
        self.num_machines = num_machines
        machine_probs = np.random.uniform(low=prob_range[0], high=prob_range[1], size=num_machines).round(2) # Win Probability of Machines
        wins = np.zeros(num_machines) #Tracks Wins for Each Machine
        losses = np.zeros(num_machines) # Tracks Losses for Each Machine
        total_reward = 0 # Tracks Overall Wins for All Rounds
        machine_played = [] # Tracks Total Plays for Each Machine

        # Iterate Through Rounds
        for i in range(0, rounds):
            probs = np.zeros(num_machines) # Store Each Machine's Draw to be Played
            
            # Iterate Through Machines
            for machine in range(0, num_machines):
                probs[machine] = random.betavariate(wins[machine] + 1, losses[machine] + 1) # Draw Based on Previous Results and Randomness

            selected_machine = np.argmax(probs) # Machine With Best Draw
            machine_played.append(selected_machine) # Collect Machine With Best Draw

            # Simulate Playing the Selected Machine, and Collect Reward
            reward = np.random.choice([0,1],1, p = [1 - machine_probs[selected_machine], machine_probs[selected_machine]])

            # Count Wins, Losses, and Reward
            if reward == 1:
                wins[selected_machine] = wins[selected_machine] + 1
            else:
                losses[selected_machine] = losses[selected_machine] + 1
            total_reward = total_reward + reward

        self.total_reward = total_reward
        self.machine_played = machine_played
        self.wins = wins
        self.losses = losses
        self.machine_probs = machine_probs
        self.rounds = rounds

        return

    def summary(self):
        # Create and Print DataFrame with Regret, Wins, Losses, and Probabilities of Machine
        regret = (max(self.machine_probs) * self.rounds) - self.total_reward
        df = pd.DataFrame(self.wins, columns=['Wins'])
        df['Losses'] = self.losses
        df['Actual Prob'] = self.machine_probs
        display(df)
        print()
        print(f"Machine {np.argmax(self.wins)} won most, with {int(self.wins[np.argmax(self.wins)])} wins")
        print(f"Machine {np.argmax(self.machine_probs)} had the highest payout")
        print(f"Total Reward: {self.total_reward[0]}")
        print(f"Total Regret: {regret[0]}")

    def hist_plot(self):
        # Plot Histogram of Total Plays For Each Machine
        plt.hist(self.machine_played, bins=self.num_machines)
        plt.title('Histogram of Machine Plays')
        plt.xlabel('Machines')
        plt.ylabel('Number of Times Selected')
        plt.show()<jupyter_output><empty_output><jupyter_text>First, we can try:*   15 bandits*   probabilities between 0.5 and 0.7*   5,000 iterationsThese numbers may not seem significant, until they are compared to A/B testing.  In that context, it is clear that almost nobody would ever test 15 variations that were expected to be almost equally likely to one another in one test, even if the sample size was 5,000.  Furthermore, each would have to be tried *too* many times for the experimenter to think the site would perform very well in the meantime.Let's see how Thompson Sampling fairs!<jupyter_code>bb = N_Bernoulli_Bandit()
bb.simulate(num_machines=15, rounds=5000, prob_range=[0.5, 0.7])
bb.summary()
bb.hist_plot()<jupyter_output><empty_output><jupyter_text>Thompson Sampling was able to find the best bandit was 5, despite the fact that 13 was only 2% less likely to pay out and the fact that half of the other options had payour probabilities within 10% of it.Furthermore, of those that were not 5, the two levers that were clearly favored were the second and third best choices, which minimized the algorithms losses in the process of learning about lever 5's dominance! Now, we can try:*   10 bandits*   probabilities between 0.2 and 0.8*   100 iterations<jupyter_code>bb = N_Bernoulli_Bandit()
bb.simulate(num_machines=10, rounds=150, prob_range=[0.2, 0.8])
bb.summary()
bb.hist_plot()<jupyter_output><empty_output><jupyter_text>Again Thompson Sampling found the top performing machine and mostly played it.  Addtionally, half of the machine probabilities were below 50% and the other half (including number 1) were above, and of the 150 pulls, only 27 were on machines below 50%.  This is not only impressive, but also saves the user a ton of money. First, we can try:*   10 bandits*   probabilities between 0.5 and 0.7*   100 iterationsLet's see if it can handle such a tall order?<jupyter_code>bb = N_Bernoulli_Bandit()
bb.simulate(num_machines=10, rounds=100, prob_range=[0.5, 0.7])
bb.summary()
bb.hist_plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07830865516715005, 'fraction_numerical': 0.016333384216150205, 'mean_word_length': 3.6043569922698526, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '50248', 'license': 'MIT', 'n_tokens_mistral': 1985, 'n_tokens_neox': 1832, 'n_words': 693, 'path': 'Using_the_N_Bernoulli_Bandit_Class.ipynb', 'repo_name': 'anthonymelson/python-rl', 'size': '6551'}