<jupyter_start><jupyter_text>Load data<jupyter_code>data_directory = "../../Downloads/ml-latest/"
movieFile = "movies.csv"
ratingFile = "ratings.csv"
df_rating = pd.read_csv(data_directory + ratingFile, dtype = {'timestamp': 'Int64'})
df_movie = pd.read_csv(data_directory + movieFile)
df_rating.head()
df_movie.head()<jupyter_output><empty_output><jupyter_text>Features we're interested in  Basic info<jupyter_code>print(f"Number of raters: %d" % (len(np.unique(df_rating['userId']))))
print("Number of movies: %d" % (len(df_movie)))
print("Number of ratings: %d" % (len(df_rating)))<jupyter_output>Number of raters: 283228
Number of movies: 58098
Number of ratings: 27753444<jupyter_text>How many ratings does a movie have?<jupyter_code>movieIds = np.unique(df_rating['movieId'])
numRatingsPerMovie = [len(df_rating[df_rating['movieId'] == movieId]) for movieId in movieIds]
print("mean number of ratings per movie: %d" % (np.mean(numRatingsPerMovie)))
print("median number of ratings per movie: %d" % (np.median(numRatingsPerMovie)))
bins = list(np.linspace(0, 20000, 20)) + list(np.linspace(20000, 100000, 4))
plt.hist(numRatingsPerMovie, bins = bins)
print(np.max(numRatingsPerMovie))
plt.scatter(movieIds, numRatingsPerMovie)
for threshold in [10, 30, 100, 500]:
    print(f"%d has fewer than {threshold} ratings" % len([i for i in numRatingsPerMovie if i < threshold]))
movieWithLessThan10Ratings = [movieIds[i] for i in range(len(movieIds)) if numRatingsPerMovie[i] < 10]
plt.scatter(movieWithLessThan10Ratings, [i for i in numRatingsPerMovie if i < 10])<jupyter_output><empty_output><jupyter_text>How many ratings does a rater provide?<jupyter_code>allRaters = np.unique(df_rating['userId'])
ratingsPerRater = {rater: [] for rater in allRaters}
for i in range(len(df_rating)):
    row = df_rating.iloc[i]
    ratingsPerRater[row['userId']].append(row['rating'])
numRatingsPerRater = [len(ratingsPerRater[rater]) for rater in allRaters]
print("mean number of ratings per rater: %d" % (np.mean(numRatingsPerRater)))
print("median number of ratings per rater: %d" % (np.median(numRatingsPerRater)))
bins = list(np.linspace(0, 2000, 20))
plt.hist(numRatingsPerRater, bins = bins)<jupyter_output>mean number of ratings per rater: 97
median number of ratings per rater: 30<jupyter_text>Are raters 'harsh' or 'nice'?<jupyter_code>meanRatingPerRater = [np.mean(ratingsPerRater[rater]) for rater in allRaters]
print("mean of mean rating per rater: %d" % (np.mean(meanRatingPerRater)))
print("median of mean rating per rater: %d" % (np.median(meanRatingPerRater)))
plt.hist(meanRatingPerRater)
plt.title("distribution of mean rating for a user")<jupyter_output>mean of mean rating per rater: 3
median of mean rating per rater: 3<jupyter_text>Are raters 'conservative' or 'liberal'?<jupyter_code>scaleOfRatingPerRater = [np.max(ratingsPerRater[rater]) - np.min(ratingsPerRater[rater]) for rater in allRaters]
print("mean of rating scale per rater: %d" % (np.mean(scaleOfRatingPerRater)))
print("median of rating scale per rater: %d" % (np.median(scaleOfRatingPerRater)))
plt.hist(scaleOfRatingPerRater)
plt.title("distribution of rating scale for a user")<jupyter_output>mean of rating scale per rater: 3
median of rating scale per rater: 3<jupyter_text>Gaussian Normalization or Decoupling Normalization?  How does Decoupling Normalization work? Suppose a rater R gives an Item a rating of r prob(R favor Item) = percentage(rating < r) + percentage(rating = r) / 2 It's basically how many items that he/she rated are rated lower than r. It actually uses order of ratings to replace ratings.<jupyter_code>POSSIBLE_RATINGS = [0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5]
def getProb(ratings, r):
    p1 = np.sum([rating < r for rating in ratings]) / len(ratings)
    p2 = np.sum([rating == r for rating in ratings]) / len(ratings)
    return p1 + p2 / 2

def getDecoupledRatings(ratings):
    prob = {r : getProb(ratings, r) for r in POSSIBLE_RATINGS}
    decoupledRatings = [prob[rating] for rating in ratings]
    return decoupledRatings

def getGaussianNormalizedRatings(ratings):
    mean = np.mean(ratings)
    std = np.std(ratings)
    ratings = np.array(ratings)
    return (ratings - mean) / std
        
gaussianRatingPerRater = {rater: getGaussianNormalizedRatings(ratingsPerRater[rater]) for rater in allRaters}
decoupledRatingPerrater = {rater: getDecoupledRatings(ratingsPerRater[rater]) for rater in allRaters}
indexRater = 3
rater = allRaters[indexRater]
decouple = getGaussianNormalizedRatings(np.sort(decoupledRatingPerrater[rater]))
gaussian = np.sort(gaussianRatingPerRater[rater])

fig = plt.figure(figsize = (20, 10))
plt.scatter(range(len(decouple)), decouple, s = 20, label = "decouple")
plt.scatter(range(len(gaussian)), gaussian, s = 20, label = "gaussian")
plt.legend(fontsize = 20)
plt.ylabel("normalized rating", fontsize = 20)
plt.xlabel("index of rating", fontsize = 20)
plt.xticks(fontsize = 20)
plt.yticks(fontsize = 20)
plt.title("side-by-side comparison of two normalizing methods on an example", fontsize = 20)
meanDecoupledRatingPerRater = [np.mean(decoupledRatingPerrater[rater]) for rater in allRaters]
print("mean of mean decoupled rating per rater: %.2f" % (np.mean(meanDecoupledRatingPerRater)))
print("median of mean decoupled rating per rater: %.2f" % (np.median(meanDecoupledRatingPerRater)))<jupyter_output>mean of mean decoupled rating per rater: 0.50
median of mean decoupled rating per rater: 0.50<jupyter_text>Observation on two normalizing method for our specific case (0.5 ~ 5 movie ratings)Decoupling method differentiates ratings more at positive ratings\Gaussian method differentiates ratings more at negative ratings Based on overall distribution of ratings (majority of ratings reside in higher range[3, 5]), decoupling method should serve better at differentiating users' ratings. Two papers comparing their performances This paper\"A Study of Methods for Normalizing User Ratings in Collaborative Filtering"\https://www.cs.purdue.edu/homes/lsi/sigir04-cf-norm.pdf \compares these two normalization methods for single-criteria collaborative filtering on a small datasetThis paper\"IMPROVING ACCURACY OF MULTI-CRITERIA COLLABORATIVE FILTERING BY NORMALIZING USER RATINGS"\https://pdfs.semanticscholar.org/0a38/aa813f16540ba2eaa3eda3a08f7c3814e079.pdf \compares these two normalization methods for multi-criteria collaborative filtering on a larger dataset Both concludes that Decoupling Normalization yields better results.  Normalize the ratings dataset<jupyter_code>otherInfosPerRater = {rater: [] for rater in allRaters}
for i in range(len(df_rating)):
    row = df_rating.iloc[i]
    otherInfosPerRater[row['userId']].append((row['movieId'], row['timestamp']))
ratingDatasetFile = 'rating_full_dataset_normalized.csv'
with open(data_directory + ratingDatasetFile, 'w', newline='') as out_csv:
    writer = csv.writer(out_csv, delimiter=',')
    writer.writerow(['rater_id', 'movie_id', 'rating', 'timestamp'])

%%time
total = len(allRaters)
count = 0
with open(data_directory + ratingDatasetFile, 'a', newline='') as out_csv:
    writer = csv.writer(out_csv, delimiter=',')
    for rater in allRaters:
        if len(otherInfosPerRater[rater]) != len(decoupledRatingPerrater[rater]):
            print(f"error! rater: {rater} len_info {len(otherInfosPerRater[rater])} len_decoupled {len(decoupledRatingPerrater[rater])}")

        for i in range(len(otherInfosPerRater[rater])):
            rating = decoupledRatingPerrater[rater][i]
            movieId = otherInfosPerRater[rater][i][0]
            timestamp = otherInfosPerRater[rater][i][1]
            writer.writerow([rater, movieId, rating - 0.5, timestamp])
    
        count += 1
        if count % 100 == 0:
            print(f'processed {count} raters, {total - count} left')<jupyter_output>processed 100 raters, 283128 left
processed 200 raters, 283028 left
processed 300 raters, 282928 left
processed 400 raters, 282828 left
processed 500 raters, 282728 left
processed 600 raters, 282628 left
processed 700 raters, 282528 left
processed 800 raters, 282428 left
processed 900 raters, 282328 left
processed 1000 raters, 282228 left
processed 1100 raters, 282128 left
processed 1200 raters, 282028 left
processed 1300 raters, 281928 left
processed 1400 raters, 281828 left
processed 1500 raters, 281728 left
processed 1600 raters, 281628 left
processed 1700 raters, 281528 left
processed 1800 raters, 281428 left
processed 1900 raters, 281328 left
processed 2000 raters, 281228 left
processed 2100 raters, 281128 left
processed 2200 raters, 281028 left
processed 2300 raters, 280928 left
processed 2400 raters, 280828 left
processed 2500 raters, 280728 left
processed 2600 raters, 280628 left
processed 2700 raters, 280528 left
processed 2800 raters, 280428 left
processed 2900 raters, 280328[...]
{'fraction_non_alphanumeric': 0.10716312861845839, 'fraction_numerical': 0.050630037461686914, 'mean_word_length': 5.909803921568628, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 27, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '561732', 'license': 'MIT', 'n_tokens_mistral': 3408, 'n_tokens_neox': 2908, 'n_words': 843, 'path': 'movieLens-data-analysis.ipynb', 'repo_name': 'yunxiaoli2017/27M-movieLens-dataset-processing', 'size': '8809'}