<jupyter_start><jupyter_text>Collaborative filtering example<jupyter_code>user,item,title = 'userId','movieId','title'<jupyter_output><empty_output><jupyter_text>Untar the data at url `URLs.ML_SAMPLE`.<jupyter_code>path = untar_data(URLs.ML_SAMPLE)<jupyter_output><empty_output><jupyter_text>Read the csv at `path/'ratings.csv'` to variable `ratings`, and show the first few rows.<jupyter_code>ratings = pd.read_csv(path/'ratings.csv')<jupyter_output><empty_output><jupyter_text>Create a `CollabDataBunch` from the df, using the ratings dataframe and a random seed of 42.<jupyter_code>data = CollabDataBunch.from_df(ratings, seed=42)<jupyter_output><empty_output><jupyter_text>Set a variable `y_range` as a 2-element array of [0, 5.5].<jupyter_code>y_range = [0, 5.5]<jupyter_output><empty_output><jupyter_text>Create a `collab_learner` from the data bunch, with 50 factors and the y_range created above.<jupyter_code>learn = collab_learner(data, n_factors=50, y_range=y_range)<jupyter_output><empty_output><jupyter_text>Fit this learner for a cycle (lr = 5e-3).<jupyter_code>learn.fit_one_cycle(1, 5e-3)<jupyter_output><empty_output><jupyter_text>Movielens 100k Create a new variable `path` set to the result of `Config.data_path()/'ml-100k'`.<jupyter_code># ! wget http://files.grouplens.org/datasets/movielens/ml-100k.zip -P {Config.data_path()} && unzip {Config.data_path()}/'ml-100k.zip' -d {Config.data_path()}
path = Config.data_path()/'ml-100k'
(Config.data_path()/'ml-100k').exists()<jupyter_output><empty_output><jupyter_text>Create another ratings dataframe, but this time:- the path should be `path/'u.data'`- the delimiter should be `\t`- there should be no header- the column names should be `userId`, `movieId`, `rating`, and `timestamp`    Why do you need these extra specifications, as compared to the original dataframe? We need the extra options to get the dataframe into the format we want.<jupyter_code>path.ls()
(path/'u.data').exists()
ratings = pd.read_csv(path/'u.data', delimiter='\t', header=None, names=['userId', 'movieId', 'rating', 'timestamp'])
ratings.head()<jupyter_output><empty_output><jupyter_text>Create a movies CSV, with:- path = `path/'u.item'`- delimiter='|'- encoding='latin-1'- no header- names = ['movieId', 'title', 'date', 'N', 'url', 'g0'-'g18']Show the first few rows of this dataframe.<jupyter_code>movies = pd.read_csv(path/'u.item', delimiter='|', encoding='latin-1', header=None, names=['movieId', 'title', 'date', 'N', 'url', *['g%d' % i for i in range(19)]])
path/'u.item'
movies.head()<jupyter_output><empty_output><jupyter_text>Take a look at the length of the ratings dataframe.<jupyter_code>ratings.shape[0]<jupyter_output><empty_output><jupyter_text>Merge the ratings and the movie dataframes into one big one that has columns:- userId- movieId- rating- timestamp- titlePrint out the head of this dataframe.<jupyter_code>path
full = ratings.merge(movies, on='movieId').loc[:, ['userId', 'movieId', 'rating', 'timestamp', 'title']]
full.head()<jupyter_output><empty_output><jupyter_text>Create a `CollabDataBunch` from this dataframe, with random seed 42, valid_pct 0.1, item_name 'movieId'.<jupyter_code>cdb = CollabDataBunch.from_df(full, seed=42, valid_pct=0.1, item_name='title')<jupyter_output><empty_output><jupyter_text>Show a batch from this dataframe.<jupyter_code>cdb.show_batch()<jupyter_output><empty_output><jupyter_text>Set `y_range` to [0, 5.5] once again.<jupyter_code>y_range = [0, 5.5]<jupyter_output><empty_output><jupyter_text>Create a collab_learner from the data bunch above, with 40 factors, using the y_range from above, with weight decay 1e-1.<jupyter_code>learn = collab_learner(cdb, n_factors=40, y_range=y_range, wd=1e-1)<jupyter_output><empty_output><jupyter_text>Find an appropriate learning rate.<jupyter_code>learn.lr_find()
learn.recorder.plot(skip_end=15)<jupyter_output><empty_output><jupyter_text>If everything is correct so far, your learning rate should be somewhere around 5e-3. Fit a 5-epoch cycle with your learning rate.<jupyter_code>learn.fit_one_cycle(5, 5e-3)<jupyter_output><empty_output><jupyter_text>The best losses should be in the low 0.8's.  Save the model under the name `dotprod`.<jupyter_code>learn.save('dotprod')<jupyter_output><empty_output><jupyter_text>Interpretation  Setup Load the `dotprod` model from above.<jupyter_code>learn.load('dotprod')<jupyter_output><empty_output><jupyter_text>Group the `rating_movie` dataset by title, and get a count of the `rating` field. Sort the values descending, and return the titles of the top 1000 rated movies.<jupyter_code>full.groupby('title').count().sort_values('rating', ascending=False).index.values[:1000]<jupyter_output><empty_output><jupyter_text>Movie bias Get the biases for the top movies using `learn.bias`. Hint: what should `is_item` be if we want movie biases?  What does the `item` variable tell us from the beginning of this notebook? `is_item` tells our collaborative filtering model how to distinguish between the fields representing the "items" in our data set, i.e. the things being rated, and the users rating them.<jupyter_code>top_rated = full.groupby('title').count().sort_values('rating', ascending=False).index.values[:1000]
# .index.sort_values(ascending=False)[:1000]
top_rated[:10]
movie_bias = learn.bias(top_rated, is_item=True)<jupyter_output><empty_output><jupyter_text>Get the average movie rating by title. Then put together a list of 3-tuples, each with the bias of the movie, the title of the movie, and the mean rating of the movie, for each of the top movies. Call this `movie_ratings`.<jupyter_code>avg_rating = full.groupby('title').mean()['rating']
avg_rating.head()
movie_ratings = [(bias, title, avg_rating.loc[title]) for bias, title in zip(movie_bias, top_rated)]
movie_ratings<jupyter_output><empty_output><jupyter_text>Create a function `item0` that grabs o[0] for some item o.<jupyter_code>item0 = lambda x: x[0]<jupyter_output><empty_output><jupyter_text>Sort the movie ratings by this function and print out the top 15 to get the worst movies.<jupyter_code>sorted(movie_ratings, key=item0)[:15]<jupyter_output><empty_output><jupyter_text>Sort the movie ratings by the reverse of this function to get the best movies.<jupyter_code>sorted(movie_ratings, key=item0, reverse=True)[:15]<jupyter_output><empty_output><jupyter_text>Movie weights Get the movie weights using `learn.weight`. Call this `movie_w`. Print out the shape.<jupyter_code>movie_w = learn.weight(top_rated)
movie_w.shape<jupyter_output><empty_output><jupyter_text>Get the first three principal components of `movie_w`. Call this `movie_pca`. Print out the shape.<jupyter_code>movie_pca = movie_w.pca(3)
movie_pca.shape<jupyter_output><empty_output><jupyter_text>Set the first three factors of `movie_pca` to variables `fac0`, `fac1`, `fac2`. Create a list of 2-tuples of the factor value and the title. Call this `movie_comp`.<jupyter_code>fac0, fac1, fac2 = movie_pca[:,0], movie_pca[:,1], movie_pca[:,2]
fac0<jupyter_output><empty_output><jupyter_text>Sort the factor/title tuple by the factor value descending and print out the top 10.<jupyter_code>sorted([(fac0[idx], title) for idx,title in enumerate(top_rated)], key=item0, reverse=True)[:10]<jupyter_output><empty_output><jupyter_text>Do the same, but not descending, and print out the first 10.<jupyter_code>sorted([(fac0[idx], title) for idx,title in enumerate(top_rated)], key=item0)[:10]<jupyter_output><empty_output><jupyter_text>Do the same process both descending and ascending for factor 1.<jupyter_code>sorted([(fac1[idx], title) for idx,title in enumerate(top_rated)], key=item0, reverse=True)[:10]
sorted([(fac1[idx], title) for idx,title in enumerate(top_rated)], key=item0)[:10]<jupyter_output><empty_output><jupyter_text>Get 50 random movies. Set X and Y to `fac0` and `fac2`. Create a pyplot figure with a scatterplot for these. Label each with the title and make each a different color.<jupyter_code>idx
idx = np.random.permutation(50)
X = [float(fac0[i]) for i in idx]
Y = [float(fac1[i]) for i in idx]
X[:10], Y[:10]
len(X), len(Y)
top_rated[idx]
from cycler import cycler
fig, ax = plt.subplots()
fig.set_size_inches(15,12)
col_cycler = cycler(color=['b', 'g', 'r', 'c', 'm', 'y', 'k'])
# col_cycler = cycler(color=['red'])
ax.set_prop_cycle(col_cycler)
for e,txt in enumerate(top_rated[idx]):
    ax.scatter(X[e], Y[e])
    ax.annotate(txt, (X[e], Y[e]))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13374363830039057, 'fraction_numerical': 0.018700437921647533, 'mean_word_length': 7.543983822042467, 'pattern_counts': {'":': 0, '<': 133, '<?xml version=': 0, '>': 133, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '638866', 'license': 'Apache-2.0', 'n_tokens_mistral': 3122, 'n_tokens_neox': 2983, 'n_words': 757, 'path': 'nbs/dl1/lesson4-collab-attempt.ipynb', 'repo_name': 'rforgione/course-v3', 'size': '8449'}