<jupyter_start><jupyter_text>References- https://www.codementor.io/jadianes/building-a-recommender-with-apache-spark-python-example-app-part1-du1083qbw  Load MovieLens data<jupyter_code>complete_dataset_url = 'http://files.grouplens.org/datasets/movielens/ml-latest.zip'
small_dataset_url = 'http://files.grouplens.org/datasets/movielens/ml-latest-small.zip'
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import scipy.sparse as sparse

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Let's start with the ratings data<jupyter_code>ratings = pd.read_csv("data/ml-latest-small/ratings.csv")
ratings.head()
ratings.userId.nunique(), ratings.movieId.nunique()<jupyter_output><empty_output><jupyter_text>We have 671 unique users and 9066 unique items. Let's see how many movies each user have rated<jupyter_code>ratings = ratings[["userId", "movieId", "rating"]]
data = ratings.groupby("userId", as_index=False).agg({"movieId": 'count'})
data.head()
data.movieId.hist()
data.movieId.describe()<jupyter_output><empty_output><jupyter_text>On average, users rated 149 movies. All users have rated at least 20 movies<jupyter_code>movies_table = pd.read_csv("data/ml-latest-small/movies.csv")
movies_table.head()<jupyter_output><empty_output><jupyter_text>This data will be helpful to match movieId with the movie title Now, let's make a user item interaction matrix. A user item interaction matrix is where each user is represented by a vector of length the number of unique items. From our ratings data, we convert each row into the specific user and item interaction and 0 everywhere else<jupyter_code>users = list(np.sort(ratings.userId.unique())) # Get our unique customers
movies = list(ratings.movieId.unique()) # Get our unique products that were purchased
rating = list(ratings.rating) # All of our purchases

rows = ratings.userId.astype('category', categories = users).cat.codes 
# Get the associated row indices
cols = ratings.movieId.astype('category', categories = movies).cat.codes 
# Get the associated column indices
user_item = sparse.csr_matrix((rating, (rows, cols)), shape=(len(users), len(movies)))
matrix_size = user_item.shape[0]*user_item.shape[1] # Number of possible interactions in the matrix
num_purchases = len(user_item.nonzero()[0]) # Number of items interacted with
sparsity = 100*(1 - (1.0*num_purchases/matrix_size))
sparsity
user_item<jupyter_output><empty_output><jupyter_text>The user item matrix has 671 unique users and 9066 unique movies which checks with what we found earlier in our data exploration. This matrix is 98.35% sparse which seems too low but it should be usable for the ALS algorithm that we will use.  Where are we? This is what we did so far:- In part 0, we downloaded our data from MovieLens, did some EDA and created our user item matrix. The matrix had 671 unique users, 9066 unique movies and is 98.35% sparse- In part 1, we described 3 of the most common recommendation method: User based collaborative filtering, Item based collaborative filtering and Matrix Facotrization- In part 2, this part, we will implement Matrix Factorization through ALS and find similar movies  Matrix Factorization We want to factorize our interaction user item matrix into a user matrix and item matrix. To do that, we will use the Alternating Least Squares (ALS) algorithm to factorize the matrix. We could write our own implementation of ALS like in this post and this post, or we can use an already available, fast implementation by <NAME>. The ALS model here is from implicit and can easily be added to your python packages or with Anaconda package manager.<jupyter_code>import implicit
model = implicit.als.AlternatingLeastSquares(factors=10, 
                                             iterations=20, 
                                             regularization=0.1, 
                                             num_threads=4)
model.fit(user_item.T)<jupyter_output><empty_output><jupyter_text>Here, we called ALS with the following parameters:- 10 factors. This indicates the number of latent factors to be used- 20 iterations- 0.1 regularization. This regularization term is the lambda in the loss function- 4 threads. This code can be parallarized which make it super fast.One thing to note is that the input for the ALS model is a item user interaction matrix, so we just have to pass the transpose of our item user matrix to the model fit function  Recommending similar movies It's time to get some results. We want to find similar movies for a selected title. The implicit module offers a ready to use method that returns similar items by providing the movie index in the item user matrix. However, we need to translate that index to the movie ID in the movies table<jupyter_code>movies_table = pd.read_csv("data/ml-latest-small/movies.csv")
movies_table.head()
def similar_items(item_id, movies_table, movies, N=5):
    """
    Input
    -----
    
    item_id: int
        MovieID in the movies table
    
    movies_table: DataFrame
        DataFrame with movie ids, movie title and genre
        
    movies: np.array
        Mapping between movieID in the movies_table and id in the item user matrix
        
    N: int
        Number of similar movies to return
        
    Output
    -----
    df: DataFrame
        DataFrame with selected movie in first row and similar movies for N next rows
    """
    # Get movie user index from the mapping array
    user_item_id = movies.index(item_id)
    # Get similar movies from the ALS model
    similars = model.similar_items(user_item_id, N=N+1)    
    # ALS similar_items provides (id, score), we extract a list of ids
    l = [item[0] for item in similars]
    # Convert those ids to movieID from the mapping array
    ids = [movies[ids] for ids in l]
    # Make a dataFrame of the movieIds
    ids = pd.DataFrame(ids, columns=['movieId'])
    # Add movie title and genres by joining with the movies table
    recommendation = pd.merge(ids, movies_table, on='movieId', how='left')
    
    return recommendation<jupyter_output><empty_output><jupyter_text>Let's try it!Let's see what similar movies do we get for a James Bond Movie: Golden Eye<jupyter_code>df = similar_items(10, movies_table, movies, 5)
df<jupyter_output><empty_output><jupyter_text>Interesting recommendations. One thing to notice is that all recommended movies are also in the Action genre. Remember that there was no indication to the ALS algorithm about movies genres. Let's try another example<jupyter_code>df = similar_items(500, movies_table, movies, 5)
df<jupyter_output><empty_output><jupyter_text>Selected movie is a comedy movie and so are the recommendations. Another interesting thing to note is that recommended movies are in the same time frame (90s).<jupyter_code>df = similar_items(1, movies_table, movies, 5)
df<jupyter_output><empty_output><jupyter_text>This is a case where the recommendations are not relevant. Recommending Silence of the Lambs for a user that just watched Toy Story does not seem as a good idea.  Make it fancy So far, the recommendations are displayed in a DataFrame. Let's make it fancy by showing the movie posters instead of just titles. To do that we will download movies metadata that I found on Kaggle. We will need the following data:- movies_metadata.csv- links<jupyter_code>metadata = pd.read_csv('data/movies_metadata.csv')
metadata.head(2)<jupyter_output><empty_output><jupyter_text>From this metadata file we only need the imdb_id and poster path.<jupyter_code>image_data = metadata[['imdb_id', 'poster_path']]
image_data.head()<jupyter_output><empty_output><jupyter_text>We want to merge this column with the movies table. Therefore, we need the links file to map between imdb id and movieId<jupyter_code>links = pd.read_csv("data/links.csv")
links.head()
links = links[['movieId', 'imdbId']]<jupyter_output><empty_output><jupyter_text>Merging the ids will be done in 2 steps:- First merge the poster path with the mapping links- Then merge with movies_tableBut first we need to remove missing imdb ids and extract the integer ID<jupyter_code>image_data = image_data[~ image_data.imdb_id.isnull()]
def app(x):
    try:
        return int(x[2:])
    except ValueError:
        print x
image_data['imdbId'] = image_data.imdb_id.apply(app)
image_data = image_data[~ image_data.imdbId.isnull()]
image_data.imdbId = image_data.imdbId.astype(int)
image_data = image_data[['imdbId', 'poster_path']]
image_data.head()
posters = pd.merge(image_data, links, on='imdbId', how='left')

posters = posters[['movieId', 'poster_path']]

posters = posters[~ posters.movieId.isnull()]

posters.movieId = posters.movieId.astype(int)

posters.head()
movies_table = pd.merge(movies_table, posters, on='movieId', how='left')
movies_table.head()<jupyter_output><empty_output><jupyter_text>Now that we have the poster path, we need to download them from a website. One way to do it is to use the TMDB API to get movie posters. However, we will have to make an account on their website, apply to use the API and wait for approval. We don't have that time, so we'll improvise.All movie posters can be accessed throught a base URL plus the movie poster path that we got, and using HTML module for python we can display them directly in Jupyter Notebook.<jupyter_code>from IPython.display import HTML
from IPython.display import display

def display_recommendations(df):
    
    images = ''
    for ref in df.poster_path:
            if ref != '':
                link = 'http://image.tmdb.org/t/p/w185/' + ref
                images += "<img style='width: 120px; margin: 0px; \
                  float: left; border: 1px solid black;' src='%s' />" \
              % link
    display(HTML(images))
df = similar_items(500, movies_table, movies, 5)
display_recommendations(df)<jupyter_output><empty_output><jupyter_text>Put all of it into one small method<jupyter_code>def similar_and_display(item_id, movies_table, movies, N=5):
    
    df = similar_items(item_id, movies_table, movies, N=N)
    
    display_recommendations(df)
similar_and_display(10, movies_table, movies, 5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0813288643533123, 'fraction_numerical': 0.009858044164037854, 'mean_word_length': 4.4164442071542975, 'pattern_counts': {'":': 1, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '120539', 'license': 'MIT', 'n_tokens_mistral': 3086, 'n_tokens_neox': 2945, 'n_words': 1127, 'path': 'notebooks/similarMovies.ipynb', 'repo_name': 'maviator/metflix', 'size': '10154'}