<jupyter_start><jupyter_text>Collaborative filtering> Using the fastai library for collaborative filtering.<jupyter_code>from fastai2.tabular.all import *
from fastai2.collab import *
# all_slow<jupyter_output><empty_output><jupyter_text>This tutorial highlights how quickly build a `Learner` and train a model on collaborative filtering tasks.  Training a model For this tutorial, we will use the [Movielens 100k data dataset](https://grouplens.org/datasets/movielens/100k/). We can download it easily and decompress it with the following function:<jupyter_code>path = untar_data(URLs.ML_100k)<jupyter_output><empty_output><jupyter_text>The main table is in `u.data`. Since it's not a proper csv, we have to specify a few things while opening it: the tab delimiter, the columns we want to keep and their names.<jupyter_code>ratings = pd.read_csv(path/'u.data', delimiter='\t', header=None,
                      usecols=(0,1,2), names=['user','movie','rating'])
ratings.head()<jupyter_output><empty_output><jupyter_text>Movie ids are not ideal to look at things, so we load the correspondence movie id to title that is in the table `u.item`:<jupyter_code>movies = pd.read_csv(path/'u.item',  delimiter='|', encoding='latin-1',
                     usecols=(0,1), names=('movie','title'), header=None)
movies.head()<jupyter_output><empty_output><jupyter_text>Next we merge it to our ratings table:<jupyter_code>ratings = ratings.merge(movies)
ratings.head()<jupyter_output><empty_output><jupyter_text>We can then build a `DataLoaders` object from this table. By default, it takes the first column for user, the second column for the item (here our movies) and the third column for the ratings. We need to change the value of `item_name` in our case, to use the titles instead of the ids:<jupyter_code>dls = CollabDataLoaders.from_df(ratings, item_name='title', bs=64)<jupyter_output><empty_output><jupyter_text>In all applications, when the data has been assembled in a `DataLoaders`, you can have a look at it with the `show_batch` method:<jupyter_code>dls.show_batch()<jupyter_output><empty_output><jupyter_text>fastai can create and train a collaborative filtering model by using `collab_learner`:<jupyter_code>learn = collab_learner(dls, n_factors=50, y_range=(0, 5.5))<jupyter_output><empty_output><jupyter_text>It uses a simple dot product model with 50 latent factors. To train it using the 1cycle policy, we just run this command:<jupyter_code>learn.fit_one_cycle(5, 5e-3, wd=0.1)<jupyter_output><empty_output><jupyter_text>Here's [some benchmarks](https://www.librec.net/release/v1.3/example.html) on the same dataset for the popular Librec system for collaborative filtering. They show best results based on RMSE of 0.91 (scroll down to the 100k dataset), which corresponds to an MSE of `0.91**2 = 0.83`. So in less than a minute, we got pretty good results!  Interpretation Let's analyze the results of our previous model. We will keep the 1000 most rated movied for this:<jupyter_code>g = ratings.groupby(title)['rating'].count()
top_movies = g.sort_values(ascending=False).index.values[:1000]
top_movies[:10]<jupyter_output><empty_output><jupyter_text>Movie bias Our model has learned one bias per movie, a unique number independent of users that can be interpreted as the intrinsic "value" of the movie. We can grab the bias of each movie in our `top_movies` list with the following conmmand:<jupyter_code>movie_bias = learn.model.bias(top_movies, is_item=True)
movie_bias.shape<jupyter_output><empty_output><jupyter_text>Let's compare those bias with the average ratings:<jupyter_code>mean_ratings = ratings.groupby(title)['rating'].mean()
movie_ratings = [(b, i, mean_ratings.loc[i]) for i,b in zip(top_movies,movie_bias)]<jupyter_output><empty_output><jupyter_text>Now let's have a look at the movies with the worst bias:<jupyter_code>item0 = lambda o:o[0]
sorted(movie_ratings, key=item0)[:15]<jupyter_output><empty_output><jupyter_text>Or the ones with the best bias:<jupyter_code>sorted(movie_ratings, key=lambda o: o[0], reverse=True)[:15]<jupyter_output><empty_output><jupyter_text>There is certainly a storng correlation!  Movie weights Now let's try to analyze the latent factors our model learned. We can grab the weights for each movie in `top_movies` the same way as we did the bias we before.<jupyter_code>movie_w = learn.model.weight(top_movies, is_item=True)
movie_w.shape<jupyter_output><empty_output><jupyter_text>Let's try a PCA to reduce the dimensions and see if we can see what the model learned:<jupyter_code>movie_pca = movie_w.pca(3)
movie_pca.shape
fac0,fac1,fac2 = movie_pca.t()
movie_comp = [(f, i) for f,i in zip(fac0, top_movies)]<jupyter_output><empty_output><jupyter_text>Here are the highest score on the first dimension:<jupyter_code>sorted(movie_comp, key=itemgetter(0), reverse=True)[:10]<jupyter_output><empty_output><jupyter_text>And the worst:<jupyter_code>sorted(movie_comp, key=itemgetter(0))[:10]<jupyter_output><empty_output><jupyter_text>Same thing for our second dimension:<jupyter_code>movie_comp = [(f, i) for f,i in zip(fac1, top_movies)]
sorted(movie_comp, key=itemgetter(0), reverse=True)[:10]
sorted(movie_comp, key=itemgetter(0))[:10]<jupyter_output><empty_output><jupyter_text>And we can even plot the movies according to their scores on those dimensions:<jupyter_code>idxs = np.random.choice(len(top_movies), 50, replace=False)
idxs = list(range(50))
X = fac0[idxs]
Y = fac2[idxs]
plt.figure(figsize=(15,15))
plt.scatter(X, Y)
for i, x, y in zip(top_movies[idxs], X, Y):
    plt.text(x,y,i, color=np.random.rand(3)*0.7, fontsize=11)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10973231696507711, 'fraction_numerical': 0.017372806240028363, 'mean_word_length': 7.002836879432624, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 82, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '28557', 'license': 'Apache-2.0', 'n_tokens_mistral': 2016, 'n_tokens_neox': 1915, 'n_words': 546, 'path': 'nbs/46_tutorial.collab.ipynb', 'repo_name': 'codescv/fastai2', 'size': '5641'}