<jupyter_start><jupyter_text>Anonymous Ratings from the Jester Online Joke Recommender System  Abstract:Ratings are real values ranging from -10.00 to +10.00 (the value "99" corresponds to "null" = "not rated").One row per userThe first column gives the number of jokes rated by that user. The next 100 columns give the ratings for jokes 01 - 100.  Dataset:    http://eigentaste.berkeley.edu/dataset/The text of the jokes can be downloaded here: jester_dataset_1_joke_texts.zip (92KB) Read the dataset(jokes.csv)Take care about the header in read_csv() as there are no column names given in the dataset.<jupyter_code>import pandas as pd
import numpy as np
data = pd.read_csv('jokes.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Consider ratings named dataframe with only first 200 rows and all columns from 1(first column is 0) of dataset<jupyter_code>ratings = data.iloc[:200,1:]<jupyter_output><empty_output><jupyter_text>Change the column indices from 0 to 99<jupyter_code>ratings.columns = range(ratings.shape[1])<jupyter_output><empty_output><jupyter_text>In the dataset, the null ratings are given as 99.00, so replace all 99.00s with 0Hint: You can use ratings.replace(, )<jupyter_code>ratings = ratings.replace(99.0, 0)
ratings<jupyter_output><empty_output><jupyter_text>Normalize the ratings using StandardScaler and save them in ratings_diff variable<jupyter_code>from sklearn.preprocessing import StandardScaler
ratings_diff = StandardScaler().fit_transform(ratings)
ratings_diff<jupyter_output><empty_output><jupyter_text>Popularity based recommendation system Find the mean for each column in ratings_diff i.e, for each joke<jupyter_code>mean_ratings = ratings_diff.mean(axis=0)<jupyter_output><empty_output><jupyter_text>Consider all the mean ratings and find the jokes with highest mean value and display the top 10 joke IDs.<jupyter_code>mean_ratings = pd.DataFrame(mean_ratings)
mean_ratings.iloc[:,0].argsort()[:-10:-1]<jupyter_output><empty_output><jupyter_text>Content based Recommendation  Context:This dataset contains ratings for ten thousand popular books. As to the source, let's say that these ratings were found on the internet. Generally, there are 100 reviews for each book, although some have less - fewer - ratings. Ratings go from one to fiveBoth book IDs and user IDs are contiguous. For books, they are 1-10000, for users, 1-53424. All users have made at least two ratings. Median number of ratings per user is 8.There are also books marked to read by the users, book metadata (author, year, etc.) and tags.Here is to recommend good book  Content:books.csv has metadata for each book (goodreads IDs, authors, title, average rating, etc.).Book metadata.best_book_id is the most popular edition for a given work. Generally it's the same as goodreads_book_id, differs occasionally.books_count is the number of editions for a given work.  Dataset:    https://www.kaggle.com/zygmunt/goodbooks-10kbooks.csv<jupyter_code>import pandas as pd

df = pd.read_csv('books.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Data pre-processing<jupyter_code># get unique books with title
books_sub = df[["book_id","title"]].copy()
books_sub.head()<jupyter_output><empty_output><jupyter_text>Vectorize Title<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
# initialize vectorizer
#vect = CountVectorizer(analyzer='word',ngram_range=(1,2),stop_words='english', min_df = 0.001)
vect = CountVectorizer(analyzer='word',ngram_range=(1,2),stop_words='english', min_df = 0.002) #min_df - rare words , max_df - most used words
#ngram_range = (1,2) - if used more than 1(value), lot of features or noise
vect.fit(books_sub['title'])
title_matrix = vect.transform(books_sub['title'])
title_matrix.shape
# Find vocabulary
features = vect.get_feature_names()
features
len(features)
What happens when we change the value of min_df value? What is this argument controlling?
What happens when we change the value of min_df value? What is this argument controlling
# confirm count of features in the corpus
word_count = [books_sub["title"].str.contains("\\b"+x+"\\b", case=False).sum() for x in features]
word_count<jupyter_output><empty_output><jupyter_text>Find Similarity Between Items<jupyter_code>from sklearn.metrics.pairwise import cosine_similarity
cosine_sim_titles = cosine_similarity(title_matrix, title_matrix)
cosine_sim_titles.shape
# get books similiar to a given title
title_id = 3
books_sub['title'].iloc[title_id]
np.argsort(cosine_sim_titles[1,])
top_n_idx = np.flip(np.argsort(cosine_sim_titles[title_id,]),axis=0)[0:10]
top_n_sim_values = cosine_sim_titles[title_id, top_n_idx]
top_n_sim_values
# find top n with values > 0
top_n_idx = top_n_idx[top_n_sim_values > 0]
# Matching books
books_sub['title'].iloc[top_n_idx]<jupyter_output><empty_output><jupyter_text>Consider the book - "To Kill a Mockingbird" (location_id - 3). What are the most similiar books? Why does "First to Kill", "Kill me if you can" etc all have same ratings?<jupyter_code>## get books similiar to a given title
title_id = 1 # make this 3
books_sub['title'].iloc[title_id]<jupyter_output><empty_output><jupyter_text>Find out what features have been considered by the vectorizer for a given title?<jupyter_code>np.where(np.squeeze(title_matrix[9247,].toarray()) > 0)
features[245]
# To find out non-zero values from the Count vectorizer matrix
title_id = 7017
feature_array = np.squeeze(title_matrix[title_id,].toarray())
idx = np.where(feature_array > 0)
idx[0]
[features[x] for x in idx[0]]
# lets wrap the above code in a function
def return_sim_books(title_id, title_matrix, vectorizer, top_n = 10):
    
    # generate sim matrix
    sim_matrix = cosine_similarity(title_matrix, title_matrix)
    features = vectorizer.get_feature_names()



    top_n_idx = np.flip(np.argsort(sim_matrix[title_id,]),axis=0)[0:top_n]
    top_n_sim_values = sim_matrix[title_id, top_n_idx]
    
    # find top n with values > 0
    top_n_idx = top_n_idx[top_n_sim_values > 0]
    scores = top_n_sim_values[top_n_sim_values > 0]
    
    
    # find features from the vectorized matrix
    sim_books_idx = books_sub['title'].iloc[top_n_idx].index
    words = []
    for book_idx in sim_books_idx:
        try:
            feature_array = np.squeeze(title_matrix[book_idx,].toarray())
        except:
            feature_array = np.squeeze(title_matrix[book_idx,])
        idx = np.where(feature_array > 0)
        words.append([" , ".join([features[i] for i in idx[0]])])
        
    # collate results
    res = pd.DataFrame({"book_title" : books_sub['title'].iloc[title_id],
           "sim_books": books_sub['title'].iloc[top_n_idx].values,"words":words,
           "scores":scores}, columns = ["book_title","sim_books","scores","words"])
    
    
    return res

vect = CountVectorizer(analyzer='word',ngram_range=(1,2),stop_words='english', min_df = 0.001)
vect.fit(books_sub['title'])
title_matrix = vect.transform(books_sub['title'])
return_sim_books(1854,title_matrix,vect,top_n=20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10154588001701886, 'fraction_numerical': 0.019004396539497943, 'mean_word_length': 5.787295476419634, 'pattern_counts': {'":': 3, '<': 53, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '279971', 'license': 'MIT', 'n_tokens_mistral': 2434, 'n_tokens_neox': 2306, 'n_words': 643, 'path': 'W1-Recommender System/W1_MentorSession/W1_CS_PB_CB_Jester_Books.ipynb', 'repo_name': 'jpthirumalai/aiml', 'size': '7051'}