<jupyter_start><jupyter_text>Bandit Feedback - Organic Best of vs Bandit Best ofIn this notebook we use the bandit signal for the first time, in the simplest possible way.  We compare how a recommender system that always makes the most popular organic product as a recommender with a recommender system that always makes the most popular bandit product.We see that there can be differences in behaviour.<jupyter_code>import gym, recogym
from recogym import env_1_args
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline

%config InlineBackend.figure_format = 'retina'
plt.rcParams['figure.figsize'] = [6, 3]

ABTestNumberOfUsers=100
NumberOfProducts=10
NumberOfSamples = 20
env_1_args['phi_var']=0.0
env_1_args['number_of_flips']=10
env_1_args['sigma_mu_organic'] = 2
env_1_args['sigma_omega']=0
env_1_args['random_seed'] = 42
env_1_args['num_products'] = NumberOfProducts
env_1_args['K'] = 5
env_1_args['number_of_flips'] = 3
env = gym.make('reco-gym-v1')
env.init_gym(env_1_args)
from recogym.agents import OrganicUserEventCounterAgent, organic_user_count_args
from recogym import Configuration

organic_counter_agent = OrganicUserEventCounterAgent(Configuration({
            **organic_user_count_args,
            **env_1_args,
            'select_randomly': True,
        }))
data = env.generate_logs(ABTestNumberOfUsers)
data[:10]<jupyter_output><empty_output><jupyter_text>Here is the data the following columns are present:* **`t`**—_`Time`_. Currently, _`Time`_ reflects the order of _`Events`_ but it does **not** represent the notion of the time in a physical sense.* **`u`**—_`User`_.* **`z`**—_`Event Type`_. There are two types of _`Events`_: _Organic_ and _Bandit_.* **`v`**—_`View`_. The column shows which _`Product`_ was shown to a _`User`_ in an _Organic_ _`Event`_.* **`a`**—_`Action`_. Currently, _`Action`_ is a _`Product`_ that was provided to a _`User`_ during a _Bandit_ _`Event`_.* **`c`**—_`Click`_. This is a _Reward_ for an _`Action`_ provided by the _`Agent`_.* **`ps`**—Probability of selecting a particular _`Action`_.**Note 1:** _`Time`_, _`User`_, _`Views`_, and _`Actions`_ **all** them start with _`0`_.**Note 2:** For any _`User`_, _Organic_ _`Event`_ _**always**_ precedes a _Bandit_ _`Event`_.  Data AnalysisLet's analyse these histograms:* _Actions per Product_* _Clicks per Product_* _Views per Product_.<jupyter_code>_, ax = plt.subplots()
ax.set_title('Histogram of Actions per Product')

ax.hist(data[data['z'] == 'bandit']['a'])
plt.show()<jupyter_output><empty_output><jupyter_text>It is evident that all _`Actions`_ are _*evenly*_ distributed.That is an expected behaviour because *all* _`Actions`_ are randomly applied for all _Bandit_ _`Events`_.  This is not typical behaviour (usually actions are personalised not random) and we will relax this assumption later in the course.<jupyter_code>clicks = np.zeros(NumberOfProducts)
bandits = data[data['z'] == 'bandit']
for product_id in range(NumberOfProducts):
    actions = bandits[bandits['a'] == product_id]
    clicks[product_id] = np.sum(actions[actions['c'] == 1]['c'])
    
print("Clicks: ", clicks)

_, ax = plt.subplots()
ax.set_title('Histogram of Clicks per Product')

ax.bar(range(NumberOfProducts), clicks)
plt.show()<jupyter_output><empty_output><jupyter_text>We can estimate a click through rate for each recommendation by dividing the number of times we obtained a click by the number of impressions.  This produces the following "bandit best of":<jupyter_code>from scipy.stats.distributions import beta

clicks = np.zeros(NumberOfProducts)
impressions = np.zeros(NumberOfProducts)
lower_errors = np.zeros(NumberOfProducts)
upper_errors = np.zeros(NumberOfProducts)
bandits = data[data['z'] == 'bandit']
for product_id in range(NumberOfProducts):
    actions = bandits[bandits['a'] == product_id]
    clicks[product_id] = np.sum(actions[actions['c'] == 1]['c'])
    impressions[product_id] = sum(actions['a']==product_id)

    # Compute expected value according to a beta distribution
    expected_value = beta.ppf(0.500, clicks[product_id] + 1, impressions[product_id] - clicks[product_id] + 1)
    # Compute a confidence interval
    lower_bound = beta.ppf(0.025, clicks[product_id] + 1, impressions[product_id] - clicks[product_id] + 1)
    upper_bound = beta.ppf(0.975, clicks[product_id] + 1, impressions[product_id] - clicks[product_id] + 1)
    # Save difference as error
    lower_errors[product_id] = expected_value - lower_bound
    upper_errors[product_id] = upper_bound - expected_value
    

_, ax = plt.subplots()
ax.set_title('Click through rate (non personalised)')

ax.errorbar(range(NumberOfProducts),
            clicks/impressions,
            yerr = (lower_errors, upper_errors),
            fmt = 'o',
            ecolor = 'darkred',
            capsize = 4)

plt.ylim([0,0.025])
plt.xlabel('Item ID')
plt.ylabel('Click-through rate')
plt.show()<jupyter_output><empty_output><jupyter_text>The non-personalised click through rate can be uses as a non-personalised agent.  This will be our first likelihood based agent.<jupyter_code>from collections import defaultdict

from recogym.agents.abstract import Agent

class GreedySingleActionAgent(Agent):
    def __init__(self, preferred_action, config = Configuration({'num_products': 10})):
        super(GreedySingleActionAgent, self).__init__(config)
        self.preferred_action = preferred_action
        
    def act(self, observation, reward, done):
        prob = np.zeros(self.config.num_products)
        prob[self.preferred_action] = 1.
        return {
            **super().act(observation, reward, done),
            **{
                'a': self.preferred_action,
                'ps': prob[self.preferred_action],
                'ps-a': prob,
            },
        }
# Initialise bandit
top_ctr_item = np.argmax(clicks/impressions)
print('The Greedy-Bandit agent will always take action {0}'.format(top_ctr_item))
greedy_bandit = GreedySingleActionAgent(top_ctr_item, Configuration(env_1_args))<jupyter_output><empty_output><jupyter_text>The ads that in a non-personalised sense attracts the most clicks is not in general the same as a product that is often viewed organically, to see this we can plot the organic views:<jupyter_code>organics = data[data['z'] == 'organic']

views = np.zeros(NumberOfProducts)
for product_id in range(NumberOfProducts):
    views[product_id] = organics[organics['v'] == product_id].shape[0]

print("Views: ", views)

_, ax = plt.subplots()
ax.set_title('Histogram of Views per Product')

ax.bar(range(NumberOfProducts), views)
plt.show()
# Plot number of organic views on x-axis and CTR on y-axis
CTR = np.array(clicks)/np.array(impressions)


_, ax = plt.subplots()
ax.set_title('CTR for number of organic views')

ax.scatter(views, CTR, marker = '.', label = 'CTR')

z = np.polyfit(views, CTR, 1)
p = np.poly1d(z)
ax.plot(views, p(views), color = 'darkgreen', label = 'Trend')

plt.xlabel('Number of organic views')
plt.ylabel('Click-through rate')

plt.gca().set_ylim([0.007,0.016])
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>While there may be a relationship between popular products and high click through rate it is certainly not guaranteed.  This fact underlies the need to use bandit feedback.  Here we will develop our final organic agent as an organic best of to hammer home this point.<jupyter_code># Initialise bandit
top_viewed_item = np.argmax(views)
print('The organic-best-of agent will always take action {0}'.format(top_viewed_item))
greedy_organic = GreedySingleActionAgent(top_viewed_item, Configuration(env_1_args))
from datetime import datetime
from recogym.evaluate_agent import verify_agents, plot_verify_agents

print('Starting A/B-testing at {0}'.format(datetime.now()))
result = verify_agents(env,
                       10000,
                       {'Greedy-Bandit': greedy_bandit,
                        'Greedy-Organic': greedy_organic})
print('Finished A/B-testing at {0}'.format(datetime.now()))
fig = plot_verify_agents(result)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11772716018816538, 'fraction_numerical': 0.010769992572418915, 'mean_word_length': 4.897080291970803, 'pattern_counts': {'":': 1, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '342040', 'license': 'Apache-2.0', 'n_tokens_mistral': 2732, 'n_tokens_neox': 2594, 'n_words': 714, 'path': 'Bandit Feedback - Organic best of vs Bandit best of.ipynb', 'repo_name': 'mindis/rnd-reco-gym', 'size': '8078'}