<jupyter_start><jupyter_text>Training a recommender system on a large datasetTrain a recommender system using a large fast.ai curated dataset<jupyter_code># imports for notebook boilerplate
!pip install -Uqq fastbook
import fastbook
from fastbook import *
from fastai.collab import *
# set up the notebook for fast.ai
fastbook.setup_book()<jupyter_output><empty_output><jupyter_text>Ingest the dataset- define the path object- define a dataframe to contain the dataset<jupyter_code># ingest the curated recommender system dataset ML_SAMPLE
path = untar_data(URLs.ML_100k)
# examine the directory structure
path.ls()
# dataset structure
'''
├── README
├── allbut.pl
├── mku.sh
├── u.data
├── u.genre
├── u.info
├── u.item
├── u.occupation
├── u.user
├── u1.base
├── u1.test
├── u2.base
├── u2.test
├── u3.base
├── u3.test
├── u4.base
├── u4.test
├── u5.base
├── u5.test
├── ua.base
├── ua.test
├── ub.base
└── ub.test
'''
# ingest the u.data file into a Pandas dataframe
# the data is this file is tab separated
df_data = pd.read_csv(path/'u.data', delimiter = '\t',header = None,names = ['userId','movieId','rating','timestamp'])
# ingest the u.item file into a dataframe
# the data in this file is vertical bar ('|') separated
df_item = pd.read_csv(path/'u.item', delimiter = '|',header = None,encoding = "ISO-8859-1")<jupyter_output><empty_output><jupyter_text>Examine the dataset<jupyter_code># examine the first few records in the df_data dataframe
df_data.head()
df_data.shape
# examine the first few records in the df_item dataframe
df_item.head()
df_item.shape<jupyter_output><empty_output><jupyter_text>Combine the dataframes- combine the two dataframes, df_data and df_item, to get a single dataframe that contains the user ID, movie ID, and movie title<jupyter_code># slice off just the first two columns of df_item and label columns
df_item = df_item.iloc[:,0:2]
df_item.columns = ['movieId','movieName']
df_item.head()
# merge the dataframes
df = pd.merge(df_data,df_item,on=['movieId'],how='left')
df.head()
# get the number of records in the dataset
df.shape
# get the count of unique values in each column of the dataset
df.nunique()
# count the number of missing values in each column of the dataset
df.isnull().sum()
df['rating'].nunique()
# defined a CollabDataLoaders object
dls=CollabDataLoaders.from_df(df,item_name='movieName',bs= 64)
dls.show_batch()<jupyter_output><empty_output><jupyter_text>Define and train the model<jupyter_code># define the model
learn=collab_learner(dls,y_range= [ 1 , 5 ] )
# train the model
learn.fit_one_cycle( 5 )
# get a selection of movies with high bias
# this cell is adapted from Howard & Gugger
movie_bias = learn.model.i_bias.weight.squeeze()
idxs = movie_bias.argsort(descending=True)[:5]
[dls.classes['movieName'][i] for i in idxs]
 
# get a selection of movies with low bias
# this cell is adapted from Howard & Gugger
idxs = movie_bias.argsort(descending=False)[:5]
[dls.classes['movieName'][i] for i in idxs]
# get the subset of ratings for one movie
df_one_movie = df[df.movieName=='Showgirls (1995)']
df_one_movie.head()
# get the average rating for this movie
df_one_movie['rating'].mean()<jupyter_output><empty_output><jupyter_text>Exercise the trained model- define a dataframe containing test data- apply the trained model to the dataframe<jupyter_code># define the test dataframe
scoring_columns = ['userId','movieId','movieName']
test_df = pd.DataFrame(columns=scoring_columns)
test_df.at[0,'userId'] = 607
test_df.at[0,'movieId'] = 242
test_df.at[0,'movieName'] = 'Kolya (1996)'
test_df.at[1,'userId'] = 607
test_df.at[1,'movieId'] = 302
test_df.at[1,'movieName'] = 'L.A. Confidential (1997)'
test_df.at[2,'userId'] = 607
test_df.at[2,'movieId'] = 375
test_df.at[2,'movieName'] = 'Showgirls (1995)'
test_df.head()
# exercise the recommender system on the test dataframe
dl = learn.dls.test_dl(test_df)
learn.get_preds(dl=dl)
learn.summary()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1306443429731101, 'fraction_numerical': 0.017757483510908167, 'mean_word_length': 5.96643109540636, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '88585', 'license': 'MIT', 'n_tokens_mistral': 1454, 'n_tokens_neox': 1416, 'n_words': 413, 'path': 'ch5/training_large_recommender_systems.ipynb', 'repo_name': 'Vega95/Deep-Learning-with-fastai-Cookbook', 'size': '3942'}