<jupyter_start><jupyter_text>WikiRecs Part 2 - Modeling and Evaluation> In this second part of the two part series, we will handle the modeling and evaluation side. We will load the saved artifacts and train 7 implicit recommender models. We will compare the performance using various evaluation metrics. We will also visualize the embeddings.- toc: true- badges: true- comments: true- categories: [Evaluation, Visualization, Implicit, GridSearch]- author: "<NAME>"- image:  Setup<jupyter_code>!git clone https://github.com/sparsh-ai/reco-wikirecs
%cd /content/reco-wikirecs/
!pip install -r requirements.txt

!pip install -q git+https://github.com/sparsh-ai/recochef.git
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

import itertools
from scipy.sparse import csr_matrix, csc_matrix, lil_matrix, coo_matrix

from recochef.datasets.wikirecs import WikiRecs

import implicit
from implicit.nearest_neighbours import bm25_weight
from implicit.nearest_neighbours import BM25Recommender

from utils import *
from wiki_pull import *
from recommenders import *
%matplotlib inline
%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Loading artifacts<jupyter_code>p2t, t2p, u2n, n2u, p2i, u2i, i2p, i2u, n2i, t2i, i2n, i2t = load_pickle('lookup_tables.pickle')
userids, pageids = load_pickle('users_and_pages.pickle')
resurface_userids, discovery_userids = load_pickle('resurface_discovery_users.pickle')
implicit_matrix = load_pickle('implicit_matrix.pickle')<jupyter_output><empty_output><jupyter_text>Test the matrix and indices<jupyter_code># Crude item to item recs by looking for items edited by the same editors (count how many editors overlap)
veditors = np.flatnonzero(implicit_matrix[t2i['Hamburger'],:].toarray())
indices =  np.flatnonzero(np.sum(implicit_matrix[:,veditors] > 0,axis=1))
totals = np.asarray(np.sum(implicit_matrix[:,veditors] > 0 ,axis=1)[indices])
sorted_order = np.argsort(totals.squeeze())

[i2t.get(i, "")  + " " + str(total[0]) for i,total in zip(indices[sorted_order],totals[sorted_order])][::-1][:10]<jupyter_output><empty_output><jupyter_text>Baseline models| Model | Type | Description || - | -:| ---:|| Popularity | Rule-based | Most popular over the past year || Recent | Rule-based | Most recently edited by this user || Frequent | Rule-based | Most frequently edited by this user in the last year || BM25 | Collaborative-filtering | Okapi BM25, a simple variation on Jaccard similarity with TF-IDF that often has much better results || ALS | Collaborative-filtering | Alternating Least Squares matrix factorization of implicit training data, with BM25 pre-scaling |  Implicit recommendation<jupyter_code>implicit_matrix = load_pickle('implicit_matrix_2021-05-28.pickle')
p2t, t2p, u2n, n2u, p2i, u2i, i2p, i2u, n2i, t2i, i2n, i2t = load_pickle('lookup_tables_2021-05-28.pickle')
bm25_matrix = bm25_weight(implicit_matrix, K1=100, B=0.25)
num_factors =200
regularization = 0.01
os.environ["OPENBLAS_NUM_THREADS"] = "1"
model = implicit.als.AlternatingLeastSquares(
    factors=num_factors, regularization=regularization
)
model.fit(bm25_matrix)
save_pickle(model,'als%d_bm25_model.pickle' % num_factors)
model = wr.load_pickle('als200_bm25_model_2021-05-28.pickle')
results = model.similar_items(t2i['Steven Universe'],20)
['%s %.4f' % (i2t[ind], score) for ind, score in results]
u = n2u["Rama"]
recommendations = model.recommend(u2i[u], bm25_matrix.tocsc(), N=1000, filter_already_liked_items=False)
[ ("*" if implicit_matrix[ind,u2i[u]]>0 else "") +
'%s %.4f' % (i2t[ind], score) + ' %d' % (implicit_matrix[ind,:]>0).sum()
 for ind, score in recommendations]<jupyter_output><empty_output><jupyter_text>Grid search results<jupyter_code>grid_search_results = wr.load_pickle("implicit_grid_search.pickle")
pd.DataFrame(grid_search_results)
pd.DataFrame([[i['num_factors'], i['regularization']] + list(i['metrics'].values()) for i in grid_search_results],
            columns = ['num_factors','regularization'] + list(grid_search_results[0]['metrics'].keys()))
grid_search_results_bm25 = wr.load_pickle("implicit_grid_search_bm25.pickle")
pd.DataFrame([[i['num_factors'], i['regularization']] + list(i['metrics'].values()) for i in grid_search_results_bm25],
            columns = ['num_factors','regularization'] + list(grid_search_results_bm25[0]['metrics'].keys()))<jupyter_output><empty_output><jupyter_text>B25 Recommendation<jupyter_code>bm25_matrix = bm25_weight(implicit_matrix, K1=20, B=1)
bm25_matrix = bm25_matrix.tocsc()
sns.distplot(implicit_matrix[implicit_matrix.nonzero()],bins = np.arange(0,100,1),kde=False)

sns.distplot(bm25_matrix[bm25_matrix.nonzero()],bins = np.arange(0,100,1),kde=False)
K1 = 100
B = 0.25
model = BM25Recommender(K1, B)
model.fit(implicit_matrix)
save_pickle(model, 'bm25_model_2021-05-28.pkl')
results = model.similar_items(t2i['<NAME>'],20)
['%s %.4f' % (i2t[ind], score) for ind, score in results]
a = ['Steven Universe 429.4746',
 'List of Steven Universe episodes 178.4544',
 'Demon Bear 128.7237',
 'Legion of Super Heroes (TV series) 128.7237',
 'The Amazing World of Gumball 126.3522',
 'Steven Universe Future 123.9198']
results = model.similar_items(t2i['Steven Universe'],20)
['%s %.4f' % (i2t[ind], score) for ind, score in results]
results = model.similar_items(t2i['<NAME>'],20)
['%s %.4f' % (i2t[ind], score) for ind, score in results]
results = model.similar_items(t2i['Hamburger'],20)
['%s %.4f' % (i2t[ind], score) for ind, score in results]
u = n2u["Rama"]
recommendations = model.recommend(u2i[u], implicit_matrix.astype(np.float32), N=1000, filter_already_liked_items=True)
[ ("*" if implicit_matrix[ind,u2i[u]]>0 else "") +
'%s %.4f' % (i2t[ind], score) 
 for ind, score in recommendations]
plt.plot([ score for i,(ind, score) in enumerate(recommendations) if implicit_matrix[ind,u2i[u]]==0])
save_pickle(model, "b25_model.pickle")
model = load_pickle("b25_model.pickle")<jupyter_output><empty_output><jupyter_text>Evaluate models  Item to item recommendation<jupyter_code>results = model.similar_items(t2i['Steven Universe'],20)
['%s %.4f' % (i2t[ind], score) for ind, score in results]<jupyter_output><empty_output><jupyter_text>User to item recommendations<jupyter_code># Check out a specific example

u = n2u["HyprMarc"]

print_user_history(clean_histories, userid=u)
u = n2u["HyprMarc"]
recommendations = model.recommend(u2i[u], implicit_matrix, N=100, filter_already_liked_items=False)
[ ("*" if implicit_matrix[ind,u2i[u]]>0 else "") +
'%s %.4f' % (i2t[ind], score) 
 for ind, score in recommendations]<jupyter_output><empty_output><jupyter_text>Visualize implicit embeddings<jupyter_code>model = load_pickle('als150_model.pickle')
# Only plot the ones with over 3 entries
indices = np.squeeze(np.asarray(np.sum(implicit_matrix[nonzero,:],axis=1))) > 3

indices = nonzero[indices]
len(indices)
# Visualize  the collaborative filtering item vectors, embedding into 2D space with UMAP
# nonzero = np.flatnonzero(implicit_matrix.sum(axis=1))
# indices = nonzero[::100]
embedding = umap.UMAP().fit_transform(model.item_factors[indices,:])
plt.figure(figsize=(10,10))
plt.plot(embedding[:,0], embedding[:,1],'.')
# _ = plt.axis('square')<jupyter_output><empty_output><jupyter_text>Visualize actors in the embeddings space<jupyter_code>edit_counts = np.squeeze(np.asarray(np.sum(implicit_matrix[indices,:],axis=1)))
log_edit_counts = np.log10(np.squeeze(np.asarray(np.sum(implicit_matrix[indices,:],axis=1))))

emb_df = pd.DataFrame({'dim1':embedding[:,0].squeeze(), 
                       'dim2':embedding[:,1].squeeze(),
                       'title':[i2t[i] for i in indices],
                       'edit_count':edit_counts,
                       'log_edit_count':log_edit_counts
                       })
actors = ['<NAME>',
'<NAME>',
'<NAME>',
'<NAME>',
'<NAME> (actor)',
'<NAME>',
'<NAME>',
'<NAME>',
'<NAME>',
'<NAME>',
'<NAME>',
'<NAME>',
'<NAME>',
'<NAME>']
actor_indices = [t2i[a] for a in actors]
edit_counts = np.squeeze(np.asarray(np.sum(implicit_matrix[actor_indices,:],axis=1)))
log_edit_counts = np.log10(np.squeeze(np.asarray(np.sum(implicit_matrix[actor_indices,:],axis=1))))
embedding = umap.UMAP().fit_transform(model.item_factors[actor_indices,:])
emb_df = pd.DataFrame({'dim1':embedding[:,0].squeeze(), 
                       'dim2':embedding[:,1].squeeze(),
                       'title':[i2t[i] for i in actor_indices],
                       'edit_count':edit_counts,
                       'log_edit_count':log_edit_counts
                       })
key = np.zeros(len(actors))
key[:8] = 1
fig = px.scatter(data_frame=emb_df,
                 x='dim1',
                 y='dim2', 
                 hover_name='title',
                 color=key,
                 hover_data=['edit_count'])
fig.update_layout(
    autosize=False,
    width=600,
    height=600,)
fig.show()
# Full embedding plotly interactive visualization

emb_df = pd.DataFrame({'dim1':embedding[:,0].squeeze(), 
                       'dim2':embedding[:,1].squeeze(),
                       'title':[i2t[i] for i in indices],
                       'edit_count':edit_counts,
                       'log_edit_count':log_edit_counts
                       })

fig = px.scatter(data_frame=emb_df,
                 x='dim1',
                 y='dim2', 
                 hover_name='title',
                 color='log_edit_count',
                 hover_data=['edit_count'])
fig.update_layout(
    autosize=False,
    width=600,
    height=600,)
fig.show()<jupyter_output><empty_output><jupyter_text>Evaluate on test set<jupyter_code># Load the edit histories in the training set and the test set
histories_train = feather.read_feather('histories_train_2021-05-28.feather')
histories_test = feather.read_feather('histories_test_2021-05-28.feather')
histories_dev = feather.read_feather('histories_dev_2021-05-28.feather')

implicit_matrix = wr.load_pickle('implicit_matrix_2021-05-28.pickle')
p2t, t2p, u2n, n2u, p2i, u2i, i2p, i2u, n2i, t2i, i2n, i2t = wr.load_pickle('lookup_tables_2021-05-28.pickle')

userids, pageids = wr.load_pickle('users_and_pages_2021-05-28.pickle')

resurface_userids, discovery_userids   = wr.load_pickle('resurface_discovery_users_2021-05-28.pickle')

results = {}
display_recs_with_history(
    recs,
    userids[:100],
    histories_test,
    histories_train,
    p2t,
    u2n,
    recs_to_display=5,
    hist_to_display=10,
)
import utils as wr
import numpy as np
from tqdm.auto import tqdm
import itertools
import pandas as pd
from implicit.nearest_neighbours import BM25Recommender


class Recommender(object):
    def __init__(self):
        raise NotImplementedError

    def recommend(self, userid=None, username=None, N=10):
        raise NotImplementedError

    def recommend_all(self, userids, num_recs, **kwargs):
        recs = {}
        with tqdm(total=len(userids), leave=True) as progress:
            for u in userids:
                recs[u] = self.recommend(userid=u, N=num_recs, **kwargs)
                progress.update(1)

        return recs


class PopularityRecommender(Recommender):
    def __init__(self, interactions):
        with wr.Timer("Building popularity table"):
            self.editors_per_page = (
                interactions.drop_duplicates(subset=["TITLE", "USER"])
                .groupby(["ITEMID", "TITLE"])
                .count()
                .USER.sort_values(ascending=False)
            )

    def recommend(self, N=10, userid=None, user=None):
        return self.editors_per_page.iloc[:N].index.get_level_values(0).values


class MostRecentRecommender(Recommender):
    """
    Recommend the most recently edited pages by the user in reverse chronological
    order. When those run out, go to most popular
    """

    def __init__(self, interactions):
        with wr.Timer("Building popularity table"):
            self.editors_per_page = (
                interactions.drop_duplicates(subset=["TITLE", "USER"])
                .groupby(["ITEMID", "TITLE"])
                .count()
                .USER.sort_values(ascending=False)
            )

    def all_recent_only(self, N=10, userids=None, interactions=None):
        recents = {}

        with tqdm(total=len(userids), leave=True) as progress:
            for u in userids:
                is_user_row = interactions.USERID == u
                recents[u] = (
                    interactions[is_user_row]
                    .drop_duplicates(subset=["ITEMID"])
                    .iloc[:N]
                    .ITEMID.values
                )
                progress.update(1)
        return recents

    def recommend(self, N=10, userid=None, user=None, interactions=None):
        if user is not None:
            is_user_row = interactions.user == user
        elif userid is not None:
            is_user_row = interactions.USERID == userid
        else:
            raise ValueError("Either user or userid must be non-null")

        deduped_pages = interactions[is_user_row].drop_duplicates(subset=["pageid"])
        if len(deduped_pages) == 1:
            recs = []
        else:
            # Don't take the most recent, because this dataset strips out repeated instance
            recs = deduped_pages.iloc[1:N].pageid.values

        # If we've run out of recs, fill the rest with the most popular entries
        if len(recs) < N:
            recs = np.concatenate(
                [
                    recs,
                    self.editors_per_page.iloc[: (N - len(recs))]
                    .index.get_level_values(0)
                    .values,
                ]
            )
        return recs


class MostFrequentRecommender(Recommender):
    """
    Recommend the most frequently edited pages by the user. When those run out, go to most popular
    """

    def __init__(self, interactions):
        with wr.Timer("Building popularity table"):
            self.editors_per_page = (
                interactions.drop_duplicates(subset=["TITLE", "USER"])
                .groupby(["pageid", "title"])
                .count()
                .USER.sort_values(ascending=False)
            )

    def recommend(self, N=10, userid=None, user=None, interactions=None):
        if user is not None:
            is_user_row = interactions.USER == user
        elif userid is not None:
            is_user_row = interactions.ITEMID == userid
        else:
            raise ValueError("Either user or userid must be non-null")

        recs = (
            interactions[is_user_row]
            .groupby("ITEMID")
            .USER.count()
            .sort_values(ascending=False)
            .index[:N]
            .values
        )

        # If we've run out of recs, fill the rest with the most popular entries
        if len(recs) < N:
            recs = np.concatenate(
                [
                    recs,
                    self.editors_per_page.iloc[: (N - len(recs))]
                    .index.get_level_values(0)
                    .values,
                ]
            )
        return recs


class ImplicitCollaborativeRecommender(Recommender):
    def __init__(self, model, implicit_matrix):
        self.model = model
        self.implicit_matrix = implicit_matrix

    def recommend(
        self,
        N=10,
        userid=None,
        user=None,
        u2i=None,
        n2i=None,
        i2p=None,
        filter_already_liked_items=False,
    ):
        if user is not None:
            user_index = n2i[user]
        elif userid is not None:
            user_index = u2i[userid]
        else:
            raise ValueError("Either user or userid must be non-null")

        recs_indices = self.model.recommend(
            user_index,
            self.implicit_matrix,
            N,
            filter_already_liked_items=filter_already_liked_items,
        )
        recs = [i2p[a[0]] for a in recs_indices]

        return recs

    def recommend_all(self, userids, num_recs, i2p, filter_already_liked_items=True):
        all_recs = self.model.recommend_all(
            self.implicit_matrix.T,
            num_recs,
            filter_already_liked_items=filter_already_liked_items,
        )
        recs = {
            userid: [i2p[i] for i in all_recs[i, :]] for i, userid in enumerate(userids)
        }

        return recs


class MyBM25Recommender(Recommender):
    def __init__(self, model, implicit_matrix):
        self.model = model

        self.implicit_matrix = implicit_matrix

    def recommend(
        self,
        N=10,
        filter_already_liked_items=True,
        userid=None,
        user=None,
        u2i=None,
        n2i=None,
        i2p=None,
    ):
        if user is not None:
            user_index = n2i[user]
        elif userid is not None:
            user_index = u2i[userid]
        else:
            raise ValueError("Either user or userid must be non-null")

        recs_indices = self.model.recommend(
            user_index,
            self.implicit_matrix.astype(np.float32),
            N,
            filter_already_liked_items=filter_already_liked_items,
        )
        recs = [i2p[a[0]] for a in recs_indices]

        if len(recs) <= 20:
            recs = recs + [recs[-1]] * (20 - len(recs))

        return recs


class JaccardRecommender(Recommender):
    def __init__(self, implicit_matrix, p2i, t2i, i2t, i2p, n2i, u2i, i2u):
        self.implicit_matrix = implicit_matrix
        self.p2i = p2i
        self.t2i = t2i
        self.i2t = i2t
        self.i2p = i2p
        self.n2i = n2i
        self.i2p = i2p
        self.u2i = u2i
        self.i2u = i2u

    def jaccard_multiple(self, page_indices, exclude_index=None):
        X = self.implicit_matrix.astype(bool).astype(int)
        if exclude_index is None:
            intrsct = X.dot(X[page_indices, :].T)
            totals = X[page_indices, :].sum(axis=1).T + X.sum(axis=1)
        else:
            use_indices = np.full(X.shape[1], True)
            use_indices[exclude_index] = False
            # print(X[:, use_indices].shape)
            # print(X[page_indices, :][:, use_indices].T.shape)

            intrsct = X[:, use_indices].dot(X[page_indices, :][:, use_indices].T)
            totals = X[page_indices, :][:, use_indices].sum(axis=1).T + X[
                :, use_indices
            ].sum(axis=1)

        return intrsct / (totals - intrsct)

    def recommend(
        self,
        N=10,
        userid=None,
        user=None,
        num_lookpage_pages=None,
        recent_pages_dict=None,
        interactions=None,
    ):
        if user is not None:
            user_index = self.n2i[user]
        elif userid is not None:
            user_index = self.u2i[userid]
        else:
            raise ValueError("Either user or userid must be non-null")

        recent_pages = recent_pages_dict[self.i2u[user_index]][:num_lookpage_pages]

        user_page_indices = [self.p2i[p] for p in recent_pages]
        d = self.jaccard_multiple(user_page_indices, exclude_index=user_index)

        d = np.nan_to_num(d)
        d[d == 1] = np.nan

        mean_jaccard = np.nanmean(d, axis=1).A.squeeze()
        order = np.argsort(mean_jaccard)[::-1]
        return [self.i2p[o] for o in order[:N]]

    def item_to_item(self, N=10, title=None, pageid=None):
        if title is not None:
            page_index = self.t2i.get(title, None)
        elif pageid is not None:
            page_index = self.p2i.get(pageid, None)
        else:
            raise ValueError("Either title or pageid must be non-null")

        if page_index is None:
            raise ValueError(
                "Page {} not found".format(pageid if title is None else title)
            )

        target_page_editors = np.flatnonzero(
            self.implicit_matrix[page_index, :].toarray()
        )
        # print("target_page_editors {}".format(target_page_editors))

        num_target_editors = len(target_page_editors)

        edited_indices = np.flatnonzero(
            np.sum(self.implicit_matrix[:, target_page_editors] > 0, axis=1)
        )

        # print("edited_indices {}".format(edited_indices))

        num_shared_editors = np.asarray(
            np.sum(self.implicit_matrix[:, target_page_editors] > 0, axis=1)[
                edited_indices
            ]
        ).squeeze()

        # print("num_shared_editors {}".format(num_shared_editors))

        num_item_editors = np.asarray(
            np.sum(self.implicit_matrix[edited_indices, :] > 0, axis=1)
        ).squeeze()

        # print("num_item_editors {}".format(num_item_editors))
        # print("Type num_item_editors {}".format(type(num_item_editors)))
        # print("num_item_editors dims {}".format(num_item_editors.shape))

        jaccard_scores = (
            num_shared_editors.astype(float)
            / ((num_target_editors + num_item_editors) - num_shared_editors)
        ).squeeze()

        # print("jaccard_scores {}".format(jaccard_scores))

        sorted_order = np.argsort(jaccard_scores)
        sorted_order = sorted_order.squeeze()

        rec_indices = edited_indices.squeeze()[sorted_order][::-1]
        sorted_scores = jaccard_scores.squeeze()[sorted_order][::-1]
        sorted_num_shared_editors = num_shared_editors.squeeze()[sorted_order][::-1]
        sorted_num_item_editors = num_item_editors.squeeze()[sorted_order][::-1]

        if title is None:
            return list(
                zip(
                    [self.i2p[i] for i in rec_indices[:N]],
                    sorted_scores[:N],
                    sorted_num_shared_editors[:N],
                    sorted_num_item_editors[:N],
                )
            )
        else:
            return list(
                zip(
                    [self.i2t[i] for i in rec_indices[:N]],
                    sorted_scores[:N],
                    sorted_num_shared_editors[:N],
                    sorted_num_item_editors[:N],
                )
            )


class InterleaveRecommender(Recommender):
    """
    Recommend for users by interleaving recs from multiple lists. When there is
    duplicates keeping only the first instance.
    """

    def __init__(self):
        pass

    def recommend_all(self, N=10, recs_list=[]):
        """
        Args:
            N (int): Number of recs to return
            recs_list: Array of recs, which are ordered lists of pageids in a dict keyed by a userid
        Returns:
            dict: Recommendations, as a list of pageids keyed by userid
        """

        def merge_page_lists(page_lists):
            return pd.unique(list(itertools.chain(*zip(*page_lists))))

        return {
            userid: merge_page_lists([recs.get(userid, []) for recs in recs_list])[:N]
            for userid in recs_list[0]
        }
results = {}<jupyter_output><empty_output><jupyter_text>Most popular<jupyter_code>%%time
K=20
rec_name = "Popularity"

prec = PopularityRecommender(histories_train)
precs = prec.recommend_all(userids, K)
# wr.save_pickle(precs, "../" + rec_name +"_recs.pickle")
results[rec_name] = get_recs_metrics(histories_dev, precs, K, discovery_userids, resurface_userids, implicit_matrix, i2p, u2i)

results[rec_name]<jupyter_output><empty_output><jupyter_text>Most recent<jupyter_code>%%time
# Most recent
K=20
rrec = recommenders.MostRecentRecommender(histories_train)
rrecs = rrec.recommend_all(userids, K, interactions=histories_train)
rec_name = "Recent"
wr.save_pickle(rrecs, "../" + rec_name +"_recs.pickle")
len(resurface_userids)
results ={}
results[rec_name] = wr.get_recs_metrics(
    histories_dev, rrecs, K, discovery_userids, resurface_userids, implicit_matrix, i2p, u2i)
results[rec_name]<jupyter_output><empty_output><jupyter_text>Most frequent<jupyter_code>%%time
# Sorted by frequency of edits
K=20
frec = recommenders.MostFrequentRecommender(histories_train)
frecs = frec.recommend_all(userids, K, interactions=histories_train)
rec_name = "Frequent"
wr.save_pickle(frecs, "../" + rec_name +"_recs.pickle")
results[rec_name] = wr.get_recs_metrics(
    histories_dev, frecs, K, discovery_userids, resurface_userids, implicit_matrix, i2p, u2i)
results[rec_name]<jupyter_output><empty_output><jupyter_text>BM25<jupyter_code>%%time
K=20
brec = recommenders.MyBM25Recommender(model, implicit_matrix)
brecs = brec.recommend_all(userids, K, u2i=u2i, n2i=n2i, i2p=i2p, filter_already_liked_items=False)
rec_name = "bm25"
wr.save_pickle(brecs, "../" + rec_name +"_recs.pickle")
# filter_already_liked_items = False
results[rec_name] = wr.get_recs_metrics(
    histories_dev, brecs, K, discovery_userids, resurface_userids, implicit_matrix, i2p, u2i)
results[rec_name]
# filter_already_liked_items = True
rec_name = "bm25_filtered"
brecs_filtered = brec.recommend_all(userids, K, u2i=u2i, n2i=n2i, i2p=i2p, filter_already_liked_items=True)
wr.save_pickle(brecs_filtered, "../" + rec_name +"_recs.pickle")
results[rec_name] = wr.get_recs_metrics(
    histories_dev, recs['bm25_filtered'], K, discovery_userids, resurface_userids, implicit_matrix, i2p, u2i)
results[rec_name]
results[rec_name] = wr.get_recs_metrics(
    histories_dev, recs['bm25_filtered'], K, discovery_userids, resurface_userids, implicit_matrix, i2p, u2i)
results[rec_name]<jupyter_output><empty_output><jupyter_text>ALS Implicit collaborative filtering<jupyter_code>model_als = wr.load_pickle('../als200_bm25_model_2021-05-28.pickle')
%%time
rec_name = "als"
K=20
irec = recommenders.ImplicitCollaborativeRecommender(model_als, bm25_matrix.tocsc())
irecs = irec.recommend_all(userids, K, i2p=i2p, filter_already_liked_items=False)
wr.save_pickle(irecs, "../" + rec_name +"_recs.pickle")
results[rec_name] = wr.get_recs_metrics(
    histories_dev, irecs, K, discovery_userids, resurface_userids, bm25_matrix.tocsc(), i2p, u2i)
results[rec_name]
rec_name = "als_filtered"
K=20
irec = recommenders.ImplicitCollaborativeRecommender(model_als, bm25_matrix.tocsc())
irecs_filtered = irec.recommend_all(userids, K, i2p=i2p, filter_already_liked_items=True)
results[rec_name] = wr.get_recs_metrics(
    histories_dev, irecs_filtered, K, discovery_userids, resurface_userids, bm25_matrix.tocsc(), i2p, u2i)
results[rec_name]
wr.save_pickle(irecs_filtered, "../" + rec_name +"_recs.pickle")
show(pd.DataFrame(results).T)<jupyter_output><empty_output><jupyter_text>Jaccard<jupyter_code>%%time
# Sorted by Jaccard
K=20
rrec = recommenders.MostRecentRecommender(histories_train)
recent_pages_dict = rrec.all_recent_only(K, userids,  interactions=histories_train)
jrec = recommenders.JaccardRecommender(implicit_matrix, p2i=p2i, t2i=t2i, i2t=i2t, i2p=i2p, n2i=n2i, u2i=u2i, i2u=i2u)
jrecs = jrec.recommend_all(userids, 
                                   K, 
                                   num_lookpage_pages=1, 
                                   recent_pages_dict=recent_pages_dict, 
                                   interactions=histories_train)
wr.save_pickle(jrecs,"jaccard-1_recs.pickle")
rec_name = "Jaccard"
results[rec_name] = wr.get_recs_metrics(
    histories_dev, jrecs, K, discovery_userids, resurface_userids, implicit_matrix, i2p, u2i)
results[rec_name]
wr.display_recs_with_history(
    jrecs,
    userids[:30],
    histories_test,
    histories_train,
    p2t,
    u2n,
    recs_to_display=5,
    hist_to_display=10,
)
%%time
# Sorted by Jaccard
K=5
jrec = recommenders.JaccardRecommender(implicit_matrix, p2i=p2i, t2i=t2i, i2t=i2t, i2p=i2p, n2i=n2i, u2i=u2i, i2u=i2u)
jrecs = jrec.recommend_all(userids[:1000], 
                                   10, 
                                   num_lookpage_pages=50, 
                                   recent_pages_dict=recent_pages_dict, 
                                   interactions=histories_train)
print("Jaccard")
print("Recall @ %d: %.1f%%" % (K, 100*wr.recall(histories_test, jrecs, K)))
print("Prop resurfaced: %.1f%%" % (100*wr.prop_resurface(jrecs, K, implicit_matrix, i2p, u2i)))
print("Recall @ %d (discovery): %.1f%%" % (K, 100*wr.recall(histories_test, jrecs, K, userid_subset=discovery_userids)))
print("Recall @ %d (resurface): %.1f%%" % (K, 100*wr.recall(histories_test, jrecs, K, userid_subset=resurface_userids)))<jupyter_output>Recall @ 5: 0.0%
Prop resurfaced: 22.0%
Recall @ 5 (discovery): 0.0%
Recall @ 5 (resurface): 0.0%<jupyter_text>Interleaved<jupyter_code>recs.keys()
# Interleaved jaccard and recent
K=20
rec_name = "Interleaved"
print(rec_name)
intrec = recommenders.InterleaveRecommender()
intrecs = intrec.recommend_all(K, [recs['Recent'], recs['bm25_filtered']])

wr.save_pickle(intrecs, "../" + rec_name +"_recs.pickle")
results[rec_name] = wr.get_recs_metrics(
    histories_dev, intrecs, K, discovery_userids, resurface_userids, implicit_matrix, i2p, u2i)
results[rec_name]<jupyter_output><empty_output><jupyter_text>Report on evaluations results  Hard coded metrics<jupyter_code>results = {}
results["Popularity"] = {'recall': 0.16187274312040842,
 'ndcg': 0.0005356797596941751,
 'resurfaced': 0.6213422985929523,
 'recall_discover': 0.11947959996459864,
 'recall_resurface': 0.2624396388830569,
 'ndcg_discover': 0.000410354483750028,
 'ndcg_resurface': 0.0008329819416998272}
results["Recent"] = {'recall': 22.618602913709378,
 'ndcg': 0.14306080818547054,
 'resurfaced': 71.13808990163118,
 'recall_discover': 0.03982653332153288,
 'recall_resurface': 76.18097837497375,
 'ndcg_discover': 0.00011494775493754298,
 'ndcg_resurface': 0.4821633227780786}
results["Frequent"] = {'recall': 20.834889802017184,
 'ndcg': 0.11356953338215306,
 'resurfaced': 76.10353629684971,
 'recall_discover': 0.035401362952473675,
 'recall_resurface': 70.17635943732941,
 'ndcg_discover': 9.90570471847343e-05,
 'ndcg_resurface': 0.38274923359395385}
results["ALS"] = {'recall': 5.488108579255385,
 'ndcg': 0.026193145556306998,
 'resurfaced': 16.251556468683848,
 'recall_discover': 1.146119125586335,
 'recall_resurface': 15.788368675204703,
 'ndcg_discover': 0.004817135435898367,
 'ndcg_resurface': 0.0769022655123215}
results["ALS_filtered"] = {'recall': 0.9027518366330469,
 'ndcg': 0.003856703716094881,
 'resurfaced': 0.0,
 'recall_discover': 1.2832994070271706,
 'recall_resurface': 0.0,
 'ndcg_discover': 0.005482465270193466,
 'ndcg_resurface': 0.0}
results["BM25"] = {'recall': 18.945336819823186,
 'ndcg': 0.1015175508656068,
 'resurfaced': 74.0469742248786,
 'recall_discover': 1.3939286662536507,
 'recall_resurface': 60.581566239764854,
 'ndcg_discover': 0.004204510293040833,
 'ndcg_resurface': 0.332367864833573}
results["BM25_filtered"] = {'recall': 1.8148424853691942,
 'ndcg': 0.008622285155255174,
 'resurfaced': 0.14848711243929774,
 'recall_discover': 2.522347110363749,
 'recall_resurface': 0.1364686122191896,
 'ndcg_discover': 0.011740495141426633,
 'ndcg_resurface': 0.0012251290280766518}
results["Interleaved"] = {'recall': 21.382766778732414,
 'ndcg': 0.12924273396038563,
 'resurfaced': 42.478676379031256,
 'recall_discover': 1.8364457031595716,
 'recall_resurface': 67.75141717404996,
 'ndcg_discover': 0.006943981897312752,
 'ndcg_resurface': 0.4193652616867473}
results_df = pd.DataFrame(results).T

results_df.reset_index(inplace=True)<jupyter_output><empty_output><jupyter_text>Table of results<jupyter_code>results_df<jupyter_output><empty_output><jupyter_text>FIG Table for post<jupyter_code>def scatter_text(x, y, text_column, data, title, xlabel, ylabel):
    """Scatter plot with country codes on the x y coordinates
       Based on this answer: https://stackoverflow.com/a/54789170/2641825"""
    
    # Create the scatter plot
    p1 = sns.scatterplot(x, y, data=data, size = 8, legend=False)
    # Add text besides each point
    for line in range(0,data.shape[0]):
         p1.text(data[x][line]+0.01, data[y][line], 
                 data[text_column][line], horizontalalignment='left', 
                 size='medium', color='black', weight='semibold')
    # Set title and axis labels
    plt.title(title)
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    return p1


def highlight_max(s):
    '''
    highlight the maximum in a Series yellow.
    '''
    is_max = s == s.max()
    return ['background-color: yellow' if v else '' for v in is_max]



results_df.sort_values("recall", ascending=False).style.apply(highlight_max, subset=["recall",
                                                                                    "ndcg",
                                                                                    "resurfaced",
                                                                                    "recall_discover",
                                                                                    "recall_resurface",
                                                                                    "ndcg_discover",
                                                                                    "ndcg_resurface",]).format({"recall": "{:.1f}%", 
                                             "ndcg": "{:.3f}",
                                             "resurfaced": "{:.1f}%", 
                                             "recall_discover": "{:.1f}%", 
                                              "recall_resurface": "{:.1f}%", 
                                            "ndcg_discover": "{:.3f}",
                                              "ndcg_resurface": "{:.3f}",
                                             })
colnames = ["Recommender", "Recall@20", "nDCG@20","Resurfaced","Recall@20 discovery","Recall@20 resurface","nDCG@20 discovery","nDCG@20 resurface"]
#apply(highlight_max, subset=colnames[1:]).
results_df.columns = colnames
results_df.sort_values("Recall@20", ascending=False).style.\
    format({"Recall@20": "{:.1f}%", 
             "nDCG@20": "{:.3f}",
             "Resurfaced": "{:.1f}%", 
             "Recall@20 discovery": "{:.1f}%", 
             "Recall@20 resurface": "{:.1f}%", 
             "nDCG@20 discovery": "{:.3f}",
             "nDCG@20 resurface": "{:.3f}",
             })<jupyter_output><empty_output><jupyter_text>Scatter plots (resurface vs discover)<jupyter_code>fig = px.scatter(data_frame=results_df,
                 x='ndcg_discover',
                 y='ndcg_resurface',
                hover_name='index')
#                  hover_name='title',)
fig.show()
fig = px.scatter(data_frame=results_df,
                 x='recall_discover',
                 y='recall_resurface',
                hover_name='index')
#                  hover_name='title',)
fig.show()<jupyter_output><empty_output><jupyter_text>FIG Scatterplot for post<jupyter_code>x = 2*[results_df.loc[results_df.Recommender == "Interleaved","Recall@20 resurface"].values[0]]
y = [0, results_df.loc[results_df.Recommender == "Interleaved","Recall@20 discovery"].values[0]]
sns.set_theme(style="darkgrid")
matplotlib.rcParams.update({'font.size': 48, 'figure.figsize':(8,5), 'legend.edgecolor':'k'})


plt.figure(figsize=(12,7))
A = results_df.loc[:,'Recall@20 discovery']
B = results_df.loc[:,'Recall@20 resurface']

x = 2*[results_df.loc[results_df.Recommender == "Interleaved","Recall@20 discovery"].values[0]]
y = [-1, results_df.loc[results_df.Recommender == "Interleaved","Recall@20 resurface"].values[0]]
plt.plot(x,y,":k")
x[0] = 0
y[0] = y[1]
# plt.rcParams.update({'font.size': 48})
plt.rc('xtick', labelsize=3)

font = {'family' : 'normal',
        'weight' : 'normal',
        'size'   : 22}

matplotlib.rc('font', **font)

plt.plot(x,y,":k")

plt.plot(A,B,'.', MarkerSize=15)


for xyz in zip(results_df.Recommender, A, B):                                       # <--
    plt.gca().annotate('%s' % xyz[0], xy=np.array(xyz[1:])+(0.05,0), textcoords='data', fontsize=18) # <--

for tick in plt.gca().xaxis.get_major_ticks():
    tick.label.set_fontsize(20)
for tick in plt.gca().yaxis.get_major_ticks():
    tick.label.set_fontsize(20)

plt.xlabel("Recall@20 discovery (%)",fontsize=20)
plt.ylabel("Recall@20 resurface (%)",fontsize=20)
plt.xlim([0,3])
plt.ylim([-2,85])
axes = plt.gca()<jupyter_output><ipython-input-60-79f665c411c3>:25: MatplotlibDeprecationWarning: Case-insensitive properties were deprecated in 3.3 and support will be removed two minor releases later
  plt.plot(A,B,'.', MarkerSize=15)<jupyter_text>Read recs in from files<jupyter_code>recommender_names = ['Popularity', 'Recent', 'Frequent', 'ALS', 'ALS_filtered', 'BM25', 'BM25_filtered', 'Interleaved']
recs = {rname:wr.load_pickle("../" + rname + "_recs.pickle") for rname in recommender_names}<jupyter_output><empty_output><jupyter_text>Recall curves<jupyter_code>histories_dev = feather.read_feather('../histories_dev_2021-05-28.feather')
plt.figure(figsize=(15,10))
for rname in recommender_names:
    recall_curve = wr.recall_curve(histories_dev, recs[rname], 20)
#     print(recall_curve[-1])
    plt.plot(recall_curve,'.-')
plt.legend(recommender_names)
plt.figure(figsize=(15,10))
for rname in recommender_names:
    recall_curve = wr.recall_curve(histories_dev, recs[rname], 20, discovery_userids)
    plt.plot(recall_curve,'.-')
plt.legend(recommender_names)
plt.figure(figsize=(15,10))
for rname in recommender_names:
    recall_curve = wr.recall_curve(histories_dev, recs[rname], 20, resurface_userids)
    plt.plot(recall_curve,'.-')
plt.legend(recommender_names)<jupyter_output><empty_output><jupyter_text>FIG Implicit vs BM25 figure<jupyter_code>sns.set_theme(style="darkgrid")
matplotlib.rcParams.update({'font.size': 18, 'figure.figsize':(8,5), 'legend.edgecolor':'k'})
plt.figure(figsize=(10,6))
for rname in ["ALS","BM25"]:
    recall_curve = wr.recall_curve(histories_dev, recs[rname], 20, discovery_userids)
    plt.plot(np.array(recall_curve)*100,'.-',markersize=12)
plt.legend( ["ALS","BM25"],title="Algorithm", fontsize=16, title_fontsize=16, facecolor="w")
plt.xlabel("@N",fontsize=20)
plt.ylabel("Discovery recall (%)",fontsize=20)
_ = plt.xticks(np.arange(0,20,2),np.arange(0,20,2)+1)
# plt.gca().legend(prop=dict(size=20))
for tick in plt.gca().xaxis.get_major_ticks():
    tick.label.set_fontsize(20)
for tick in plt.gca().yaxis.get_major_ticks():
    tick.label.set_fontsize(20)<jupyter_output><empty_output><jupyter_text>User recommendation comparison<jupyter_code>recs_subset = ["Recent","Frequent","Popularity","Implicit","bm25","interleaved"]
print("Next edit: " + histories_dev.loc[histories_dev.userid == userid].title.values[0])<jupyter_output>Next edit: Police van<jupyter_text>FIG Rama table<jupyter_code>def bold_viewed(val, viewed_pages):
    """
    Takes a scalar and returns a string with
    the css property `'color: red'` for negative
    strings, black otherwise.
    """
    weight = 'bold' if val in  viewed_pages else 'normal'
    return 'font-weight: %s' % weight

def color_target(val, target_page):
    """
    Takes a scalar and returns a string with
    the css property `'color: red'` for negative
    strings, black otherwise.
    """
    color = 'red' if val ==  target_page else 'black'
    return 'color: %s' % color

def display_user_recs_comparison(user_name, recs, recs_subset, train_set, test_set, N=20):
    userid = n2u[user_name]
    recs_table = pd.DataFrame({rec_name: [p2t[r] for r in recs[rec_name][userid][:N]] for rec_name in recs_subset})
    recs_table = recs_table.reset_index()
    recs_table.loc[:,"index"] = recs_table.loc[:,"index"]+1
    recs_table = recs_table.rename(columns={"index":""})
    viewed_pages = train_set.loc[train_set.userid == userid,["title"]].drop_duplicates(subset=["title"]).values.squeeze()
    target_page = test_set.loc[test_set.userid == userid].title.values[0]
#     print("Next edit: " + target_page)
    s = recs_table.style.applymap(bold_viewed, viewed_pages=viewed_pages).applymap(color_target, target_page=target_page)
    display(s)
recs_subset = ["Recent","Frequent","Popularity","ALS","ALS_filtered","BM25","BM25_filtered"]

display_user_recs_comparison('Rama', recs, recs_subset, histories_train, histories_dev, N=10)<jupyter_output><empty_output><jupyter_text>Other individuals tables<jupyter_code>display_user_recs_comparison('Meow', recs, recs_subset, histories_train, histories_dev, N=10)
display_user_recs_comparison('KingArti', recs, recs_subset, histories_train, histories_dev, N=10)
display_user_recs_comparison('Tulietto', recs, recs_subset, histories_train, histories_dev, N=10)
display_user_recs_comparison('Thornstrom', recs, recs_subset, histories_train, histories_dev, N=10)<jupyter_output>Next edit: List of Netflix original films (2021–)<jupyter_text>FIG Interleaved<jupyter_code>display_user_recs_comparison('Rama', recs,['Interleaved'], histories_train, histories_dev, N=10)
display_user_recs_comparison('KingArti', recs,['Interleaved'], histories_train, histories_dev, N=10)
N = 20
display(pd.DataFrame({rec_name: [p2t[r] for r in recs[rec_name][n2u['HenryXVII']]][:N] for rec_name in recs_subset}))
persons_of_interest = [
    "DoctorWho42",
    "AxelSjögren",
    "<NAME>",
    "Tulietto",
    "LipaCityPH",
    "<NAME>",
    "Thornstrom",
    "Meow",
    "HyprMarc",
    "Jampilot",
    "Rama"
]
N=10
irec_500 = recommenders.ImplicitCollaborativeRecommender(model, implicit_matrix)
irecs_poi = irec_500.recommend_all([n2u[user_name] for user_name in persons_of_interest], N, u2i=u2i, n2i=n2i, i2p=i2p)<jupyter_output><empty_output><jupyter_text>Find interesting users<jupyter_code>edited_pages = clean_histories.drop_duplicates(subset=['title','user']).groupby('user').userid.count()

edited_pages = edited_pages[edited_pages > 50]
edited_pages = edited_pages[edited_pages < 300]
clean_histories.columns
display_user_recs_comparison("Rama", recs, recs_subset, histories_train, histories_dev, N=20)
index = list(range(len(edited_pages)))
np.random.shuffle(index)

for i in index[:10]:
    user_name = edited_pages.index[i]
    print(user_name)
    display_user_recs_comparison(user_name, recs, recs_subset, histories_train, histories_dev, N=20)
    print("\n\n\n")
index = list(range(len(edited_pages)))
np.random.shuffle(index)

for i in index[:10]:
    print(edited_pages.index[i])
    display_user_recs_comparison
    wr.print_user_history(user=edited_pages.index[i],all_histories=clean_histories)
    print("\n\n\n")
sns.distplot(edited_pages,kde=False,bins=np.arange(0,2000,20))<jupyter_output><empty_output><jupyter_text>Repetition analysis<jupyter_code>import itertools
clean_histories.head()
clean_histories.iloc[:1000].values.tolist()
df = clean_histories
dict(zip(df.columns, range(len(df.columns))))
def identify_runs(df):
    d  = df.loc[:,['userid','pageid']].values.tolist()
    return [(k, len(list(g))) for k,g in itertools.groupby(d)]
%%time
runs = identify_runs(clean_histories)
lens = np.array([r[1] for r in runs])

single_edits = np.sum(lens==1)
total_edits = len(clean_histories)

print("Percent of edits that are part of a run: %.1f%%" % (100*(1-(float(single_edits)/total_edits))))

print("Percent of edits that are repetitions: %.1f%%" % (100*(1-len(runs)/total_edits)))<jupyter_output>Percent of edits that are part of a run: 48.4%
Percent of edits that are repetitions: 34.7%
{'fraction_non_alphanumeric': 0.13121170468047433, 'fraction_numerical': 0.04487104815600028, 'mean_word_length': 3.4550077841203946, 'pattern_counts': {'":': 17, '<': 150, '<?xml version=': 0, '>': 156, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 45, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '425762', 'license': 'Apache-2.0', 'n_tokens_mistral': 16458, 'n_tokens_neox': 14988, 'n_words': 2669, 'path': '_docs/nbs/wikirecs-04-modeling.ipynb', 'repo_name': 'sparsh-ai/recohut', 'size': '43067'}