<jupyter_start><jupyter_text>LikeWise A book recommender system By _<NAME>_ ------ Outline* [Introduction]()* [_Imports and Configuration_]()* [Data]()* [Modeling]()    * [Training]()    * [Generating Recommendations]() ------ Introduction Stop! Collaborate and Filter[Collaborative Filtering](https://d2l.ai/chapter_recommender-systems/recsys-intro.htmlcollaborative-filtering) (CF)> In general, CF only uses the user-item interaction data to make predictions and recommendations. ------ Imports and Configuration<jupyter_code># === General imports === #
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import os
# === fastai imports === #
from fastai.collab import *
# === Configuration === #
%matplotlib inline
pd.options.display.max_columns = 100
pd.options.display.max_rows = 200
from google.colab import drive
drive.mount('/content/drive')
# === Set up path to project dir === #
PROJECT_DIR = "/content/drive/My Drive/workshop/buildbox/likewise"<jupyter_output><empty_output><jupyter_text>------ Data> Intro to and explanation of dataset — why this dataset?The dataset used for the LikeWise recommender system is called the [UCSD Book Graph - GoodReads Datasets](https://sites.google.com/eng.ucsd.edu/ucsdbookgraph/home). For this particular model, I'll be using the "Shelves" dataset, which has interactions between users and books (ratings).The relevant columns are `user_id`, `book_id`, and `rating`. They are pretty self-explanatory, but just to be explicit, each record indicates a user rating a book that they've (presumably) read and have an opinion on.Speaking of explicit, the fact that the user _explicitly_ rates the books makes this dataset one of explicit preferences.<jupyter_code># === Load data from Drive === #
data_file = "interactions_mystery_thriller_crime_1m_sample.csv"
data_path = os.path.join(PROJECT_DIR, "assets/data", data_file)
interactions = pd.read_csv(data_path)
print(interactions.shape)
interactions.head()
# === Convert `book_id` to string === #
interactions["book_id"] = interactions["book_id"].astype("str")
# === Data types === #
interactions.dtypes
# === Convert unread "ratings" to nulls === #
interactions["rating"] = np.where(interactions["is_read"] == False, np.NaN, interactions["rating"])
interactions.head()
interactions.isnull().sum()
# === Get dfs of read/unread books === #
unread = interactions[interactions["is_read"] == False].copy()
print(f"Number of unread books: {unread.shape[0]}")

read = interactions[interactions["is_read"] == True].copy()
print(f"Number of read books: {read.shape[0]}")<jupyter_output>Number of unread books: 477996
Number of read books: 522004<jupyter_text>------ ModelingFor this model, I'll be using the FastAI collaborative filtering algorithm(s).Resources:* [fastai.collab](https://docs.fast.ai/collab.html)* [movielens recommender example](https://github.com/microsoft/recommenders/blob/master/notebooks/00_quick_start/fastai_movielens.ipynb)  Training<jupyter_code># === First, train on rated books only === #
rating_range = [0, 5]

# Create databunch
data = CollabDataBunch.from_df(
    read,
    user_name="user_id",
    item_name="book_id",
    rating_name="rating",
    valid_pct=0.2,
    seed=92
)

# === Instantiate learner === #
learn = collab_learner(data, n_factors=50, y_range=rating_range)
learn.model
data.show_batch()
# === Train! === #
learn.fit_one_cycle(3, 5e-3)
# === Export trained model === #
data_file = "01_likewise.pkl"
data_path = os.path.join(PROJECT_DIR, "assets/models", data_file)
learn.export(data_path)<jupyter_output><empty_output><jupyter_text>--- Generate Recommendations<jupyter_code># === Import the model from file === #
learner = load_learner(
    path=os.path.join(PROJECT_DIR, "assets/models"),
    file="01_likewise.pkl",
)
doc(learner.predict)
# === 3 user-shelved books that have not been reviewed === #
interactions.iloc[1]
interactions.iloc[2]
interactions.iloc[3]
# === Generate prediction for one book === #
learner.predict(interactions.iloc[2])
interactions.iloc[2]<jupyter_output><empty_output><jupyter_text>It worked! The model predicts that this user would rate this book a 3.9787<jupyter_code># === Try another unread book === #
learner.predict(interactions.iloc[3])
# === And another === #
print(unread.iloc[2345])
learner.predict(unread.iloc[2345])
# === Try one that has been rated, to see the difference === #
print(interactions.iloc[4])
learner.predict(interactions.iloc[4])<jupyter_output>user_id    8842281e1d1347389f2ab93d60773d4d
book_id                            28684704
is_read                                True
rating                                    3
Name: 4, dtype: object<jupyter_text>That prediction isn't very close. Let's try another.<jupyter_code># === Try another that has been rated === #
print(interactions.iloc[855])
learner.predict(interactions.iloc[855])<jupyter_output>user_id    4b3636a043e5c99fa27ac897ccfa1151
book_id                               66528
is_read                                True
rating                                    4
Name: 855, dtype: object<jupyter_text>That one is a lot better!<jupyter_code># === User that doesn't exist === #
interactions[interactions["user_id"] == "617ccec66dac2d1029600ed3d706e8er"]
# === User doesn't exist, book does === #
new_user_book = pd.Series(data=["617ccec66dac2d1029600ed3d706e8er", "27276292", np.NaN], index=["user_id", "book_id", "rating"])
new_user_book
# === Predict unknown user's rating === #
learner.predict(new_user_book)<jupyter_output><empty_output><jupyter_text>As can be seen in the above three cells, I created a new user and predicted what rating they would give a book. The rating it spit out was 2.7348.That seemed a little weird. So I tried it again, with a different `user_id`. It gave the same rating. I'm wondering if that's just the mean rating of that book or something.<jupyter_code># === Get all users and items that the model knows === #
total_users, total_items = learner.data.train_ds.x.classes.values()
total_users = total_users[1:]
total_items = total_items[1:]

print(total_users.shape, total_items.shape)<jupyter_output><empty_output><jupyter_text>--- More recommendations<jupyter_code># === Replace values not known to the model with NaN === #
unread.loc[~unread["user_id"].isin(total_users), "user_id"] = np.NaN
unread.loc[~unread["book_id"].isin(total_items), "book_id"] = np.NaN
# === Extract only the users and items in the training data === #
unread["user_id"] = unread.loc[unread["user_id"].isin(total_users), "user_id"]
unread["book_id"] = unread.loc[unread["book_id"].isin(total_items), "book_id"]
print(unread.shape)
unread.head()
# === Map ids to embedding ids === #
u = learner.get_idx(unread["user_id"], is_item=False)
m = learner.get_idx(unread["book_id"], is_item=True)
# === Create predictions === #
pred = learner.model.forward(u, m)
# Get all users from the test set and remove any users
# that were know in the training set

test_users = test_df[USER].unique()
test_users = np.intersect1d(test_users, total_users)
# === Cartesian product === #
from itertools import product

users_items = product(np.array(total_users), np.array(total_items))
users_items = pd.DataFrame(users_items, columns=["user_id", "book_id"])
users_items<jupyter_output><empty_output><jupyter_text>My search for how to generate recommendations led me to looking at the [`score` function](https://github.com/microsoft/recommenders/blob/master/reco_utils/recommender/fastai/fastai_utils.py) in the reco_utils module, which is used in one of the notebooks I'm referencing [here](https://github.com/microsoft/recommenders/blob/master/notebooks/00_quick_start/fastai_movielens.ipynb).I copied the `score` function into the cell below to try and use some of it to write my own recommendation function.<jupyter_code>def score(
    learner,
    test_df,
    user_col=cc.DEFAULT_USER_COL,
    item_col=cc.DEFAULT_ITEM_COL,
    prediction_col=cc.DEFAULT_PREDICTION_COL,
    top_k=None,
):
    """Score all users+items provided and reduce to top_k items per user if top_k>0
    
    Args:
        learner (obj): Model.
        test_df (pd.DataFrame): Test dataframe.
        user_col (str): User column name.
        item_col (str): Item column name.
        prediction_col (str): Prediction column name.
        top_k (int): Number of top items to recommend.
    Returns:
        pd.DataFrame: Result of recommendation 
    """
    # replace values not known to the model with NaN
    total_users, total_items = learner.data.train_ds.x.classes.values()
    test_df.loc[~test_df[user_col].isin(total_users), user_col] = np.nan
    test_df.loc[~test_df[item_col].isin(total_items), item_col] = np.nan

    # map ids to embedding ids
    u = learner.get_idx(test_df[user_col], is_item=False)
    m = learner.get_idx(test_df[item_col], is_item=True)

    # score the pytorch model
    pred = learner.model.forward(u, m)
    scores = pd.DataFrame(
        {user_col: test_df[user_col], item_col: test_df[item_col], prediction_col: pred}
    )
    scores = scores.sort_values([user_col, prediction_col], ascending=[True, False])
    if top_k is not None:
        top_scores = scores.groupby(user_col).head(top_k).reset_index(drop=True)
    else:
        top_scores = scores
    return top_scores<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1278932070190548, 'fraction_numerical': 0.019916029712563246, 'mean_word_length': 4.76302729528536, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 42, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '307760', 'license': 'MIT', 'n_tokens_mistral': 3171, 'n_tokens_neox': 3007, 'n_words': 763, 'path': 'ds/projects/likewise/notebooks/03_likewise_model.ipynb', 'repo_name': 'tobias-fyi/vela', 'size': '9296'}