<jupyter_start><jupyter_text>Collaborative Filtering in 9 Lines of CodeI was looking for some sample code for getting my hands dirty with some recommender system code, particularly around collaborative filtering. I came across [this blog post](http://tungwaiyip.info/2012/Collaborative%20Filtering.html) with the same title, which is based on the book Toby Seragan's book [Programming Collective Intelligence](http://shop.oreilly.com/product/9780596529321.do). This notebook is based on the orginal blog post. The code had to be adjusted to make it work with Python 3, and the latest version of pandas.<jupyter_code>import numpy as np; import pandas as pd; from pandas import Series, DataFrame<jupyter_output><empty_output><jupyter_text>The file contains ratings from different critics on various titles.<jupyter_code>rating = pd.read_csv('data/movie_rating.csv')
rating.head()<jupyter_output><empty_output><jupyter_text>We will first create the matrix with titles of movies as rows and critics as columns. Each cell contains the rating from the corresponding user for a rating.<jupyter_code>rp = rating.pivot_table(index=['title'], columns=['critic'], values='rating')
rp<jupyter_output><empty_output><jupyter_text>The next step is to find the similarity score between the critics. We will use _Toby_ as example, and use _Pearson correlation score_. Pandas contains the function `corrwith()` which compute the correlation. As you can see from the result below, _Toby_'s taste is similar to _Lisa Rose_ but not so much wit _Gene Seymour_.Note that we could have used some other similarity metric such as cosine similarity.<jupyter_code>rating_toby = rp['Toby']
sim_toby = rp.corrwith(rating_toby)
sim_toby<jupyter_output><empty_output><jupyter_text>To make recommendation for Toby, we calculate a rating of others weighted by the similarity. Note that we only need to calculate rating for movies Toby has not yet seen. The first line below filter out irrelevant data. It then assigns the similarity score and the weighted rating.<jupyter_code>rating_c = rating.loc[rating_toby[rating.title].isnull().values & (rating.critic != 'Toby')]
rating_c_similarity = rating_c['critic'].map(sim_toby)

rating_c = rating_c.assign(similarity=rating_c_similarity, sim_rating=rating_c_similarity * rating_c.rating)
rating_c.head()<jupyter_output><empty_output><jupyter_text>Lastly we add up the score for each title using groupby(). We also normalize the score by dividing it with the sum of the weights. Base on other critics' similarity and their rating, we have made a movie recommendation for Toby. The number matches the result of the book.<jupyter_code>recommendations = rating_c.groupby('title').apply(lambda s: s.sim_rating.sum() / s.similarity.sum())
recommendations.sort_index(ascending=False)<jupyter_output><empty_output><jupyter_text>Putting it all together:<jupyter_code>rating = pd.read_csv('data/movie_rating.csv')
rp = rating.pivot_table(index=['title'], columns=['critic'], values='rating')

rating_toby = rp['Toby']
sim_toby = rp.corrwith(rating_toby)

rating_c = rating.loc[rating_toby[rating.title].isnull().values & (rating.critic != 'Toby')]
rating_c_similarity = rating_c['critic'].map(sim_toby)
rating_c = rating_c.assign(similarity=rating_c_similarity, sim_rating=rating_c_similarity * rating_c.rating)

recommendations = rating_c.groupby('title').apply(lambda s: s.sim_rating.sum() / s.similarity.sum())
recommendations.sort_index(ascending=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09111747851002866, 'fraction_numerical': 0.006017191977077364, 'mean_word_length': 6.862612612612613, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '253950', 'license': 'MIT', 'n_tokens_mistral': 1105, 'n_tokens_neox': 1047, 'n_words': 339, 'path': 'Collaborative Filtering.ipynb', 'repo_name': 'saksham/recommender-systems', 'size': '3490'}