<jupyter_start><jupyter_text>Movie recommendation using SVDFollowing was one of Scalable Data Mining Assignments.1. Load the MovieLens data and Convert the data into a matrix (for now, let it be dense)2. (Optional: if the data is too big to handle, choose a subset of the ratings)3. Leave out some non zero entries as a test set4. Perform normalization 5. Perform SVD6. Compute the low rank ratings matrix according to the basic latent factor model7. Test performance against the test data that is left outFirst, download the data from: https://grouplens.org/datasets/movielens/  1. Load the dataset and convert the data into a matrixLoad the ratings.csv file and convert the data into a matrix R. Depending on how you load it, your movies may be rows or columns (and users vice versa). Mention that in the comment.<jupyter_code>"""
import requests
url = 'http://files.grouplens.org/datasets/movielens/ml-1m.zip'
content = requests.get(url)

# unzip the content
from io import BytesIO
from zipfile import ZipFile
f = ZipFile(BytesIO(content.content))
print(f.namelist())
f.extractall('./data')
"""<jupyter_output><empty_output><jupyter_text>Import Libraries<jupyter_code>import pandas as pd
import numpy as np
from scipy import sparse
from scipy.sparse.linalg import svds, eigs
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Read Movies names<jupyter_code>movies = pd.read_csv('./data/ml-1m/movies.dat', sep='::', header=None, usecols=[0,1,2], 
                 names = ['movieID', 'movie_name', 'Genres', 'timestamp'],engine='python', error_bad_lines=False)
# movies.to_csv('List of Movies.csv')
movies['Genre_List'] = [movies['Genres'][i].split('|') for i in range(len(movies))]
movies.head(5)<jupyter_output><empty_output><jupyter_text>Getting number of unique genres<jupyter_code>genres = np.unique([movies['Genre_List'][i][j] for i in range(len(movies)) for j in range(len(movies['Genre_List'][i]))])
print(' No of movies:  ', len(movies), '\n',
      'List of', len(genres), 'genres:', ', '.join(genres))<jupyter_output>No of movies:   3883 
 List of 18 genres: Action, Adventure, Animation, Children's, Comedy, Crime, Documentary, Drama, Fantasy, Film-Noir, Horror, Musical, Mystery, Romance, Sci-Fi, Thriller, War, Western<jupyter_text>Reading Rating Data<jupyter_code>df = pd.read_csv('./data/ml-1m/ratings.dat', sep='::', header=None, usecols=[0,1,2], 
                 names = ['userID', 'movieID', 'rating'], engine='python', error_bad_lines=False)
# df.to_csv('Base Data.csv')
p=len(df['userID'].unique());q=len(df['movieID'].unique());r=len(movies);s=len(df);t=df['rating'].mean();u=df['rating'].std()
print(f'{p} unique users have rated {q} movies out of {r} and gave {s} ratings with an average of {t:.2f} and SD of {u:.2f}')<jupyter_output>6040 unique users have rated 3706 movies out of 3883 and gave 1000209 ratings with an average of 3.58 and SD of 1.12<jupyter_text>Converting dataframe into sparse matrix for running SVD**For my matrix, movies are columns and users are rows**<jupyter_code>row = df.userID
col = df.movieID
data = df.rating
A = sparse.coo_matrix((data, (row, col)), dtype=float)
# np.savetxt("sparse A.csv", A.toarray(), delimiter=",")
row.shape, col.shape, data.shape, A.shape, type(A)<jupyter_output><empty_output><jupyter_text>There are gaps in the movie id-s - 3953 vs. 3706 unique movie id-s. The gaps must be removed. Also, since the indexing starts from 0, one extra and row and column has been added.  Mapping unique movie id-s to a running serial number<jupyter_code>keyset = sorted(df['movieID'].unique())
valset = np.arange(len(df['movieID'].unique()))
movie_mapping = dict(zip(keyset,valset))
max(keyset), min(keyset), max(valset), min(valset)<jupyter_output><empty_output><jupyter_text>Mapping unique user id-s to a running serial number<jupyter_code>keyset = sorted(df['userID'].unique())
valset = np.arange(len(df['userID'].unique()))
user_mapping = dict(zip(keyset,valset))
max(keyset), min(keyset), max(valset), min(valset)<jupyter_output><empty_output><jupyter_text>Mapping movie id-s to movie names<jupyter_code>keyset = movies['movieID']
valset = movies['movie_name']
movie_name_mapping = dict(zip(keyset,valset))<jupyter_output><empty_output><jupyter_text>Mapping movie id-s to movie genres<jupyter_code>keyset = movies['movieID']
valset = movies['Genres']
movie_genre_mapping = dict(zip(keyset,valset))<jupyter_output><empty_output><jupyter_text>Now we have to re-write the raw data in terms of the user and movie mapping above and then update the dataframe<jupyter_code>df['user_srl'] = df['userID'].replace(user_mapping, inplace=False)
df['movie_srl'] = df['movieID'].replace(movie_mapping, inplace=False)
df['movie'] = df['movieID'].replace(movie_name_mapping, inplace=False)
df['Genre'] = df['movieID'].replace(movie_genre_mapping, inplace=False)
df.head()<jupyter_output><empty_output><jupyter_text>Recreate the user-item matrix with contiguous id-s mapped from original user and movie id. **For my matrix, movies are columns and users are rows**<jupyter_code>row = df.user_srl
col = df.movie_srl
data = df.rating
R = sparse.coo_matrix((data, (row, col)), dtype=float)
# np.savetxt("sparse A.csv", A.toarray(), delimiter=",")
row.shape, col.shape, data.shape, R.shape, type(R)<jupyter_output><empty_output><jupyter_text>Run SVD on modified sparse rating matrix R, and reconstruct R_dash as an estimate of R<jupyter_code># Reconstruct SVD
from numpy import array
from numpy import diag
from numpy import dot
from numpy import zeros
from scipy.linalg import svd

U, s, VT = svd(R.toarray())
# create m x n Sigma matrix
Sigma = zeros((R.shape[0], R.shape[1]))
# populate Sigma with n x n diagonal matrix
Sigma[:R.shape[1], :R.shape[1]] = diag(s)
# reconstruct matrix
R_dash = U.dot(Sigma.dot(VT))<jupyter_output><empty_output><jupyter_text>The User and Movie Shapes<jupyter_code>VT.shape, U.shape, s.shape
# vt.T[0].sum(), u[0].sum()<jupyter_output><empty_output><jupyter_text>2. Optional: Reduce the size of the matrixIf you failed trying to run SVD with your matrix due to computational resources, reduce the size of the matrix and call the new matrix R. If you did not need to reduce the size of R, then simply set skip this step.  Ran it on 1 million rating dataset. No problem.  3. Set your test set apartRandomly choose about 2000 non-zero entries in R (2000 known ratings given by users to certain items) and create the test matrix B. Define A to be the rest of the non-zero entries. Essentially, A + B should be same as R. After the next few lines of code, you should have two matrices A and B. Note: A and B should have no common non-zero entry. The task here is to predict the zero (unknown) entries in A and test them against the available non-zero (known) entries in B.  Test data carved out of full dataset<jupyter_code>test = df.sample(n=2000, random_state=99)
row_te = test.user_srl
col_te = test.movie_srl
data_te = test.rating
p=len(test['user_srl'].unique());q=len(test['movie_srl'].unique());r=len(movies);s=len(test);t=test['rating'].mean();
u=test['rating'].std()
print(f'In the test dataset {p} unique users rated {q} movies out of total {r} and gave {s} ratings with an average of \
{t:.2f} and SD of {u:.2f}')
test.head()<jupyter_output><empty_output><jupyter_text>Creating the balance as training dataset<jupyter_code>train = df.loc[~df.index.isin(test.index), :]
row_tr = train.user_srl
col_tr = train.movie_srl
data_tr = train.rating
p=len(train['user_srl'].unique());q=len(train['movie_srl'].unique());r=len(movies);s=len(train);t=train['rating'].mean();
u=train['rating'].std()
print(f'In the train dataset {p} unique users rated {q} movies out of total {r} and gave {s} ratings with an average of \
{t:.2f} and SD of {u:.2f}')
train.head()<jupyter_output><empty_output><jupyter_text>The train and test data have very similar distribution judged from their mean and SD vs. the total data**Create train dataset for SVD**<jupyter_code>B = sparse.coo_matrix((data_te, (row_te, col_te)), dtype=float)
A = sparse.coo_matrix((data_tr, (row_tr, col_tr)), dtype=float)
# index
A.shape, (A>0).sum(), A.sum(), A.mean(), A.sum()/(A>0).sum()
# B.shape, (B>0).sum(), B.sum(), B.mean(), B.sum()/(B>0).sum()<jupyter_output><empty_output><jupyter_text>Run full SVD on sparse training matrix A amd reconstruct A_dash<jupyter_code># Reconstruct SVD
from numpy import array
from numpy import diag
from numpy import dot
from numpy import zeros
from scipy.linalg import svd
# Singular-value decomposition
U, s, VT = svd(A.toarray())
# create m x n Sigma matrix
Sigma = zeros((A.shape[0], A.shape[1]))
# populate Sigma with n x n diagonal matrix
Sigma[:A.shape[1], :A.shape[1]] = diag(s)
# reconstruct matrix
A_dash = U.dot(Sigma.dot(VT))<jupyter_output><empty_output><jupyter_text>The User and Movie Shapes<jupyter_code>VT.shape, U.shape, s.shape
# vt.T[0].sum(), u[0].sum()<jupyter_output><empty_output><jupyter_text>Compute Error vs. 2000 held out ratings in test dataframe based on un-normalized training data using A_dash<jupyter_code>error = [A_dash[test.iloc[i]['user_srl'], test.iloc[i]['movie_srl']] - test.iloc[i]['rating'] for i in range(len(test))]
rmse = np.sqrt(np.mean(np.square(error)))
print('Root mean square error of test data using un-normalized raw training data:', rmse)<jupyter_output>Root mean square error of test data using un-normalized raw training data: 3.7393849761692106<jupyter_text>Very high RMSE with reconstructed A  4. Normalize and center Please note that in A, B or R, the zero entries do not mean the corresponding ratings are zero. They are just unknown. Hence, you need to impute the missing (zero now) entries of A by the average of the known entries (row or column depending on how you have loaded your matrix) for each item. Call this matrix A1.Then, subtract the average rating given by each user from the corresponding row (or column). Call this matrix A2.  Compute Average Movie Ratings of all users who have rated the movie i.e. column averages<jupyter_code>A1 = A.toarray()
movie_avg = [A1[:,i].sum()/(A1[:,i]>0).sum() for i in range(A.shape[1])]<jupyter_output><empty_output><jupyter_text>Insert average rating of a movie in the blanks of the corresponding column<jupyter_code>for i in range(A.shape[1]):
    A1[A1[:,i]==0, i] = movie_avg[i]<jupyter_output><empty_output><jupyter_text>Compute Average Rating of each user for the movies he/she has rated i.e. row averages<jupyter_code>user_avg = A1.mean(axis=1).reshape(-1,1)
user_avg, user_avg.shape<jupyter_output><empty_output><jupyter_text>Normalized Rating of each user for each movie he/she has rated or imputed with movie average - dense matrix<jupyter_code>A2 = A1 - user_avg
# A1 = 
# A2 =<jupyter_output><empty_output><jupyter_text>5. Compute SVD Compute SVD of the matrix A2.<jupyter_code># Use your choice of variables for U, Sigma and V_transpose
# Compute SVD of A2<jupyter_output><empty_output><jupyter_text>Run full SVD on filled in training matrix A2 and reconstruct A2_dash<jupyter_code># Reconstruct SVD
from numpy import array
from numpy import diag
from numpy import dot
from numpy import zeros
from scipy.linalg import svd
# Singular-value decomposition
U, s, VT = svd(A2)
# create m x n Sigma matrix
Sigma = zeros((A2.shape[0], A2.shape[1]))
# populate Sigma with n x n diagonal matrix
Sigma[:A2.shape[1], :A2.shape[1]] = diag(s)
# reconstruct matrix
A2_dash = U.dot(Sigma.dot(VT))<jupyter_output><empty_output><jupyter_text>The User and Movie Shapes<jupyter_code>VT.shape, U.shape, s.shape
# vt.T[0].sum(), u[0].sum()<jupyter_output><empty_output><jupyter_text>Reconstruct A_dash from SVD components of A2 and evaluate with test data. Here we will have to add back the user average rating that was subtracted to get A2.  Compute Error compared 2000 rating that were held out in test dataframe<jupyter_code>error_A2 = [A2_dash[test.iloc[i]['user_srl'], test.iloc[i]['movie_srl']] + 
            user_avg[test.iloc[i]['user_srl']] -
            test.iloc[i]['rating'] for i in range(len(test))]
rmse_A2 = np.sqrt(np.mean(np.square(error_A2)))
print(f'Root mean square error of test data using full normalized A2: {rmse_A2:.6f}')<jupyter_output>Root mean square error of test data using full normalized A2: 0.977687<jupyter_text>MUCH BETTER!**Down to under 1 scale point**  6. Compute the low rank approximation Compute the low rank approximation of A2. The choice of k is up to you, but we would encourage you to try out a few different k and see which works better. Mention your experience in the readme report.  Run SVD on sparse training matrix A2, with different values of k to minimize recontruction error. We have 0.98 as baseline with full SVD reconstruction.https://machinelearningmastery.com/singular-value-decomposition-for-machine-learning/  Small function to compute reconstructed matrix for different values of k<jupyter_code>def reconfromsvd(a, k):
    u, s, vt = svds(a,k)
    # create kxk Sigma matrix
    Sigma = np.zeros((k,k))
    # populate Sigma with k diagonal elements
    np.fill_diagonal(Sigma, s)
    # reconstruct matrix
    a_dash = u.dot(Sigma.dot(vt))
    
    return a_dash<jupyter_output><empty_output><jupyter_text>Plot for optimum number of dimensions K<jupyter_code>plot_val = []
for K in np.arange(10, 310, 5):
    A2_dash = reconfromsvd(A2,K)
    error = [A2_dash[test.iloc[i]['user_srl'], test.iloc[i]['movie_srl']] +
             user_avg[test.iloc[i]['user_srl']] -
             test.iloc[i]['rating'] for i in range(len(test))]
    rmse = np.sqrt(np.mean(np.square(error)))
    val = [K,rmse]
    plot_val.append(val)
x = [plot_val[i][0] for i in range(len(plot_val))]
y = [plot_val[i][1] for i in range(len(plot_val))]
plot_val
plt.figure(figsize=(15,5))
plt.plot(x, y, 'ro')
plt.axis([0, 310, 0.895, 0.93])
# plt.xaxis([np.arange(10, 210, 10)])
# plt.yaxis([0.89, 0.92])
plt.xlabel('K=Dimensions')
plt.ylabel('Recon RMSE')
plt.title('Finding Minimum RMSE for K Low Rank Apprx of A2 on test data', fontsize=20)
plt.show()<jupyter_output><empty_output><jupyter_text>RMSE corresponding to K=40 gives the lowest value<jupyter_code>RMSE_Ak_min=plot_val[6][1];K_min=plot_val[6][0]
print(f'The lower Root mean square error, {RMSE_Ak_min:.6f} for test data comes for k = {K_min} using normalized A2')<jupyter_output>The lower Root mean square error, 0.898376 for test data comes for k = 40 using normalized A2<jupyter_text>k=40 minimizes SVD recon RMSE=0.8984 on normalized A2. Can be thought of as there exists 40 sub genres which are different combinations of 18 current ones that should provide coherent clustering of movies  7. Test the performanceCompute the root-mean-squared-error (RMSE) against the known ratings to test how the method performed, using A1 (which predicts ratings just by average known ratings) and Ak against B.The RMSE is the square root of the average squared errors between the known ratings and the predicted ratings. For our set up, consider only the pairs of (user,item) pairs (i,j) for which B[i,j] > 0. Then compute the square root of the average squared difference between A2 and B (also Ak and B) only for this set of pairs (i,j). $$RMSE(A2, B) = \sqrt{\frac{\sum_{i,j: B_{ij} > 0}(A2_{ij} - B_{ij})^2}{\sum_{i,j: B_{ij} > 0} 1}}.$$$$RMSE(Ak, B) = \sqrt{\frac{\sum_{i,j: B_{ij} > 0}(Ak_{ij} - B_{ij})^2}{\sum_{i,j: B_{ij} > 0} 1}}.$$<jupyter_code># Write your code here (several lines)
# At the end of the code, your must have two variables with the RMSEs (will not be 0.0), as given below
RMSE_A2 = rmse_A2
RMSE_Ak = RMSE_Ak_min
print("RMSE with A2 is %f." %RMSE_A2)
print("RMSE with Ak is %f." %RMSE_Ak)<jupyter_output>RMSE with A2 is 0.977687.
RMSE with Ak is 0.898376.
{'fraction_non_alphanumeric': 0.12579821200510855, 'fraction_numerical': 0.018454661558109833, 'mean_word_length': 5.583018074821354, 'pattern_counts': {'":': 0, '<': 123, '<?xml version=': 0, '>': 133, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '249242', 'license': 'MIT', 'n_tokens_mistral': 5641, 'n_tokens_neox': 5341, 'n_words': 1724, 'path': 'movie_reco_svd.ipynb', 'repo_name': 'SanjayDatta/movie-reco-with-svd', 'size': '15660'}