<jupyter_start><jupyter_text>Movie Recommendation SystemThe goal of this assignment is give you practice working with Singular Value Decomposition. Your task is implement a matrix factorization method-such as singular value decomposition (SVD). Recommender System using SVDMatrix factorization can be used to discover features underlying the interactions between two different kinds of entities. And one obvious application is to predict ratings in collaborative filtering—in other words, to recommend items to users.One advantage of employing matrix factorization for recommender systems is the fact that it can incorporate implicit feedback—information that’s not directly given but can be derived by analyzing user behavior—such as items frequently bought or viewed.Using this capability we can estimate if a user is going to like a movie that they never saw. And if that estimated rating is high, we can recommend that movie to the user, so as to provide a more personalized experience.For example, two users might give high ratings to a certain movie if they both like the actors/actresses of the movie, or if the movie is a thriller movie, which is a genre preferred by both users.Hence, if we can discover these kinds of latent features (like genre or actors and directors), we should be able to predict a rating with respect to a certain user and a certain item, because the features associated with the user should match with the features associated with the item.The data input for a recommender system can be thought of as a large matrix, with the rows indicating an entry for a customer, and the columns indicating an entry for a particular item. Let’s call this matrix $R$. Then entry $R_{i,j}$ will contain the score that customer $i$ has given to product $j$. For example if it’s a review this could be a number from 1–5, or it might just be 0–1 indicating if a user has bought an item or not. This matrix contains a lot of missing information, it’s unlikely a customer has bought every item on Amazon! Recommender systems aim to fill in this missing information, by predicting the customer score of items where the score is missing. Then recommender systems will recommend items to the customer that have the highest score. A typical example of the matrix $R$ with entries that are review values from 1–5 is given in the picture below. <NAME> Roll number: 2019121004   Imports<jupyter_code>import pandas as pd
import numpy as np
import scipy
from scipy.linalg import sqrtm<jupyter_output><empty_output><jupyter_text>DatasetThe dataset we’ll be working with is a very famous movies dataset: the ml-1m, or the [MovieLens dataset 100 k](https://grouplens.org/datasets/movielens/100k). 1. Data PreprocessingWe will begin by loading the dataset file present in the `.csv` file into pandas dataframes and visualizing the entries.<jupyter_code>data = pd.read_csv('movielens100k.csv')
data['userId'] = data['userId'].astype('str')
data['movieId'] = data['movieId'].astype('str')

users = data['userId'].unique() #list of all users
movies = data['movieId'].unique() #list of all movies

print("Number of users", len(users))
print("Number of movies", len(movies))

data.head()<jupyter_output>Number of users 718
Number of movies 8915<jupyter_text>2. Split the data into a train and test set<jupyter_code>test = pd.DataFrame(columns=data.columns)
train = pd.DataFrame(columns=data.columns)
test_ratio = 0.2 #fraction of data to be used as test set.
for u in users:
    temp = data[data['userId'] == u]
    n = len(temp)
    test_size = int(test_ratio*n)
    temp = temp.sort_values('timestamp').reset_index()
    temp.drop('index', axis=1, inplace=True)

    dummy_test = temp.iloc[n-1-test_size :]
    dummy_train = temp.iloc[: n-2-test_size]

    test = pd.concat([test, dummy_test])
    train = pd.concat([train, dummy_train])<jupyter_output><empty_output><jupyter_text>3. Create the utility matrixThe input data will now be converted to the utility matrix $(n\times m)$ where the rows of the matrix are users $n$ and the columns are the ratings for the $m$-th movie.<jupyter_code># Create_utility_matrix
def create_utility_matrix(data, formatizer = {'user':0, 'item': 1, 'value': 2}):
    """
        :param data:      Array-like, 2D, nx3
        :param formatizer:pass the formatizer
        :return:          utility matrix (n x m), n=users, m=items
    """
        
    itemField = formatizer['item']
    userField = formatizer['user']
    valueField = formatizer['value']
    userList = data.iloc[:,userField].tolist()
    itemList = data.iloc[:,itemField].tolist()
    valueList = data.iloc[:,valueField].tolist()
    users = list(set(data.iloc[:,userField]))
    items = list(set(data.iloc[:,itemField]))
    users_index = {users[i]: i for i in range(len(users))}
    pd_dict = {item: [np.nan for i in range(len(users))] for item in items}
    for i in range(0,len(data)):
        item = itemList[i]
        user = userList[i]
        value = valueList[i]
        pd_dict[item][users_index[user]] = value
    X = pd.DataFrame(pd_dict)
    X.index = users
        
    itemcols = list(X.columns)
    items_index = {itemcols[i]: i for i in range(len(itemcols))}
    # users_index gives us a mapping of user_id to index of user
    # items_index provides the same for items
    return X, users_index, items_index

utilMat, users_index, items_index = create_utility_matrix(train)<jupyter_output><empty_output><jupyter_text>Metric computationThe function rmse computes the root mean square error (RMSE) for the true and the predicted movie ratings.<jupyter_code>def rmse(true, pred):
    # this will be used to compute the root mean square error for the true and the predicted movie rating
    x = true - pred
    return sum([xi*xi for xi in x])/len(x)<jupyter_output><empty_output><jupyter_text>Code for computing SVD for the utility matrix [Write your code here]<jupyter_code>######## WRITE YOUR CODE TO COMPUTE THE SVD BELOW #############
def svd(mat, k):
    arr_p = np.ma.masked_array(mat, np.isnan(mat))
    mat = arr_p.filled(np.mean(arr_p, axis=0))
    x = np.tile(np.mean(arr_p, axis=0), (mat.shape[0],1))
    U, s, V=np.linalg.svd(mat-x, full_matrices=False)
    s=np.diag(s)
    ans = np.dot(np.dot(U[:,0:k],sqrtm(s[0:k,0:k])), np.dot(sqrtm(s[0:k,0:k]),V[0:k,:])) +x
    return ans<jupyter_output><empty_output><jupyter_text>Code for the test set [Write your code here]Write the code that computes the RMSE for the predicted ratings for the test data present in the `test` matrix.<jupyter_code>######## WRITE YOUR CODE TO COMPUTE RMSE FOR THE TEST SET BELOW #############
def rmse(true, arr_pred):
    return sum([xi*xi for xi in true - arr_pred])/len(true)


mat, users_ind, movie_arr = create_utility_matrix(train)
for i in range(6,18): 
    arr_pred = []
    out = svd(np.array(mat), k=i)
    for _,ar in test.iterrows():
        if ar['movieId'] not in movie_arr:
            arr_pred.append(np.mean(out[users_ind[ar['userId']], :]))
        else:
            arr_pred.append(out[users_ind[ar['userId']], movie_arr[ar['movieId']]])      
    err = rmse(test['rating'], arr_pred)
    print("number of features: "+str(i)+"  rsme: "+str(err))<jupyter_output>number of features: 6  rsme: 0.9634459056195747
number of features: 7  rsme: 0.9631666230809447
number of features: 8  rsme: 0.9630898543487947
number of features: 9  rsme: 0.9624551442352094
number of features: 10  rsme: 0.9632320309708742
number of features: 11  rsme: 0.9635170902573413
number of features: 12  rsme: 0.9645970994047941
number of features: 13  rsme: 0.964966608785657
number of features: 14  rsme: 0.9632251443983305
number of features: 15  rsme: 0.9626140805934346
number of features: 16  rsme: 0.9621906198648389
number of features: 17  rsme: 0.9624754027509491
{'fraction_non_alphanumeric': 0.09907240401958259, 'fraction_numerical': 0.03645967534140685, 'mean_word_length': 4.463054187192118, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '646448', 'license': 'MIT', 'n_tokens_mistral': 2569, 'n_tokens_neox': 2376, 'n_words': 915, 'path': 'HW4/2019121004.ipynb', 'repo_name': 'avani17101/SMAI-Assignments', 'size': '7767'}