<jupyter_start><jupyter_text>Likelihood based modelsThis notebook will outline the likelihood based approach to training on Bandit feedback.Although before proceeding we will study the output of the simmulator in a little more detail.<jupyter_code>from numpy.random.mtrand import RandomState
from recogym import Configuration
from recogym.agents import Agent
from sklearn.linear_model import LogisticRegression
from recogym import verify_agents
from recogym.agents import OrganicUserEventCounterAgent, organic_user_count_args
from recogym.evaluate_agent import verify_agents, plot_verify_agents



import gym, recogym
from copy import deepcopy
from recogym import env_1_args
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline

%config InlineBackend.figure_format = 'retina'
plt.rcParams['figure.figsize'] = [6, 3]

ABTestNumberOfUsers=1000
NumberOfProducts=10
NumberOfSamples = 20
env_1_args['phi_var']=0.0
env_1_args['number_of_flips']=0
env_1_args['sigma_mu_organic'] = 0.0
env_1_args['sigma_omega']=1
env_1_args['random_seed'] = 42
env_1_args['num_products'] = NumberOfProducts
env_1_args['K'] = 5
env_1_args['number_of_flips'] = 5

env = gym.make('reco-gym-v1')
env.init_gym(env_1_args)
data = deepcopy(env).generate_logs(ABTestNumberOfUsers)<jupyter_output>Organic Users: 0it [00:00, ?it/s]
Users: 100%|██████████| 1000/1000 [00:17<00:00, 57.33it/s]<jupyter_text>Logistic Regression Model  Turn Data into Features Now we are going to build a _Logistic Regression_ model.The model will predict _the probability of the click_ for the following data:* _`Views`_ is a total amount of views of a particular _`Product`_ shown during _Organic_ _`Events`_ **before** a _Bandit_ _`Event`_.* _`Action`_ is a proposed _`Product`_ at a _Bandit_ _`Event`_.For example, assume that we have _`10`_ products. In _Organic_ _`Events`_, these products  were shown to a user as follows:            Product ID        Views                0        0                1        0                2        0                3        7                4        0                5        0                6        0                7        8                8        11                9        0    When we want to know the probability of the click for _`Product`_ = _`8`_ with available amounts of _`Views`_, the input data for the model will be:_`0 0 0 7 0 0 0 0 8 11 0`_ _**`8`**_The first 10 numbers are _`Views`_ of _`Products`_ (see above), the latest one is the _`Action`_.The output will be two numbers:* $0^{th}$ index: $1 - \mathbb{P}_c(P=p|V)$.* $1^{st}$ index: $\mathbb{P}_c(P=p|V)$.Here, $\mathbb{P}_c(P=p|V)$ is the probability of the click for a _`Product`_ $p$, provided that we have _`Views`_ $V$.In all following models, an _`Action`_ will not be used as a number, but it will be decoded as a _vector_.In our current example, the _`Action`_ is _`8`_. Thus, it is encoded as:_`0 0 0 0 0 0 0 0`_ _**`1`**_ _`0`_Here,* Vector of _`Actions`_ has a size that is equal to the _*number of `Products`*_ i.e. _`10`_.* _`Action`_ _`8`_ is marked as _`1`_ (_`Action`_ starts with _`0`_).<jupyter_code>import math
import numpy as np

def build_train_data(data):
    """
    Build Train Data

        Parameters:
            data: offline experiment logs
                the data contains both Organic and Bandit Events

        Returns:
            :(outs, history, actions)
    """
    num_products = int(data.v.max() + 1)
    number_of_users = int(data.u.max()) + 1

    history = []
    actions = []
    outs = []

    for user_id in range(number_of_users):
        views = np.zeros((0, num_products))
        for _, user_datum in data[data['u'] == user_id].iterrows():
            if user_datum['z'] == 'organic':
                assert (math.isnan(user_datum['a']))
                assert (math.isnan(user_datum['c']))
                assert (not math.isnan(user_datum['v']))

                view = int(user_datum['v'])

                tmp_view = np.zeros(num_products)

                tmp_view[view] = 1

                # Append the latest view at the beginning of all views.
                views = np.append(tmp_view[np.newaxis, :], views, axis = 0)
            else:
                assert (user_datum['z'] == 'bandit')
                assert (not math.isnan(user_datum['a']))
                assert (not math.isnan(user_datum['c']))
                assert (math.isnan(user_datum['v']))

                action = int(user_datum['a'])
                action_flags = np.zeros(num_products, dtype = np.int8)
                action_flags[int(action)] = 1

                click = int(user_datum['c'])

                history.append(views.sum(0))
                actions.append(action_flags)
                outs.append(click)

    return np.array(outs), history, actions
clicks, history, actions = build_train_data(data)
data[0:27]
history[0:8]
actions[0:8]<jupyter_output><empty_output><jupyter_text>Look at the data and see how it maps into the features - which is the combination of the history and the actions and the label which is clicks.  Note that only the bandit events correspond to records in the training data.In order to do personalisation it is necessary to cross the action and history features. _Why_?  We do the simplest possible cross an element wise kronecker product.<jupyter_code>from recogym.agents import FeatureProvider

class CrossFeatureProvider(FeatureProvider):
    """Feature provider as an abstract class that defined interface of setting/getting features"""

    def __init__(self, config):
        super(CrossFeatureProvider, self).__init__(config)
        self.feature_data = None

    def observe(self, observation):
        """Consider an Organic Event for a particular user"""
        for session in observation.sessions():
            self.feature_data[session['v']] += 1

    def features(self, observation):
        """Provide feature values adjusted to a particular feature set"""
        return self.feature_data

    def reset(self):
        self.feature_data = np.zeros((self.config.num_products))


class ModelBasedAgent(Agent):
    def __init__(self, env, feature_provider, model):
        # Set environment as an attribute of Agent.
        self.env = env
        self.feature_provider = feature_provider
        self.model = model
        self.reset()

    def act(self, observation, reward, done):
        """Act method returns an action based on current observation and past history"""
        self.feature_provider.observe(observation)
        cross_features = np.kron(np.eye(env.config.num_products),self.feature_provider.features(observation))
        prob = self.model.predict_proba(cross_features)[:, 1]
        action = np.argmax(prob)
        prob = np.zeros_like(prob)
        prob[action] = 1.0
        return {
            **super().act(observation, reward, done),
            **{
                'a': action,
                'ps': 1.,
                'ps-a': prob,
            }
        }

    def reset(self):
        self.feature_provider.reset()
def build_history_agent(env_args, data):
    outs, history, actions = build_train_data(data)
    features = np.vstack([np.kron(aa,hh) for hh, aa in zip(history, actions)])

    config = Configuration(env_args)
    logreg = LogisticRegression(
        solver = 'lbfgs',
        max_iter = 5000,
        random_state = config.random_seed
    )

    log_reg_fit = logreg.fit(features, outs)

    return ModelBasedAgent(
        config,
        CrossFeatureProvider(config),
        log_reg_fit
    )


likelihood_logreg =  build_history_agent(env_1_args, data)
organic_counter_agent = OrganicUserEventCounterAgent(Configuration({
            **organic_user_count_args,
            **env_1_args,
            'select_randomly': True,
        }))
result = verify_agents(env, 5000, {'likelihood logreg': likelihood_logreg, 'organic count': organic_counter_agent})
fig = plot_verify_agents(result)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10129608657354977, 'fraction_numerical': 0.01849754624386561, 'mean_word_length': 2.877073170731707, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '28106', 'license': 'Apache-2.0', 'n_tokens_mistral': 2524, 'n_tokens_neox': 2360, 'n_words': 720, 'path': 'Bandit Feedback - Likelihood based feature engineering logistic regression.ipynb', 'repo_name': 'mindis/rnd-reco-gym', 'size': '7947'}