<jupyter_start><jupyter_text>Exploratory data analysis<jupyter_code>import pandas as pd
import csv
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Read in ratings file<jupyter_code>ratings = pd.read_csv("ratings_clean.csv",  quoting=csv.QUOTE_NONNUMERIC)<jupyter_output><empty_output><jupyter_text>The main info that I need to know for the model is the distribution of ratings for each user (rater) and for each movie.To start off with I'll examine the distribution of ratings for each user and the number of users with more than 10 and 100 ratings:<jupyter_code>df = ratings.groupby("rater_id")[["rating"]].count()
print(df[df['rating']>=10].size)
print(df[df['rating']>=100].size)
df.columns = ["Number of ratings for users (raters)"]
df.hist()
plt.xlabel("Total ratings for a user")
plt.ylabel("Number of users")
df.describe()<jupyter_output>13009
1619<jupyter_text>Here we can see the ratings are sparse -- most users have few ratings. However, there are still 13009 users with 10 or more ratings and 1619 users with 100 or more ratings. The average is 13.6 ratings per user, and the median is 2 -- the distribution is extemely right skewed. Next, I'll look at the mean rating for each user:<jupyter_code>df = ratings.groupby("rater_id")[["rating"]].mean()
df.columns = ["Mean ratings for users"]
df.hist()
plt.xlabel("Rating (10 point scale)")
plt.ylabel("Number of users")
df.describe()<jupyter_output><empty_output><jupyter_text>The mean rating is 7.9 and median is 8. Most users tend to rate movies favorably on average.Next I'll examine the distrubutions for movies, starting with the number of ratings per movie:<jupyter_code>df = ratings.groupby("movie_id")[["rating"]].count()
df.columns = ["Number of ratings for movies"]
df.hist()
plt.xlabel("Total ratings for a movie")
plt.ylabel("Number of movies")
df.describe()<jupyter_output><empty_output><jupyter_text>Another extremely right skewed distribution, with a mean of 23.6 and median of 2. Clearly, most movies have a few ratings, and a small minority of movies have more than 1000 reviews.Now let's look at mean rating per movie:<jupyter_code>df = ratings.groupby("movie_id")[["rating"]].mean()
df.columns = ["Mean ratings for movies"]
df.hist()
plt.xlabel("Mean rating (out of 10 points)")
plt.ylabel("Number of movies")
df.describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11323155216284987, 'fraction_numerical': 0.02205258693808312, 'mean_word_length': 6.303405572755418, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '635560', 'license': 'MIT', 'n_tokens_mistral': 807, 'n_tokens_neox': 733, 'n_words': 262, 'path': 'EDA/EDA.ipynb', 'repo_name': 'iafinn/twitter-mov-rec', 'size': '2358'}