<jupyter_start><jupyter_text>Ratings dataset<jupyter_code>data = pd.read_csv(rating_file)
print('shape: ', data.shape)
data.head()<jupyter_output><empty_output><jupyter_text>1. Outliers There are some people that review a large number of times<jupyter_code>x = data['userId'].value_counts();
users = x.index[:]
counts = x.iloc[:]
counts
print('There are {} unique number of reviewers'.format(users.shape))
#filter out people with 2 standard deviations away from the mean rating per person
def reject_outliers(data, m=2):
    return data[abs(data - np.mean(data)) < m * np.std(data)]
filtered = reject_outliers(counts, m= 2)
reject_outliers(counts, m= 2)[:10]
filtered.shape<jupyter_output><empty_output><jupyter_text>Distribution of ratings per reviewer The majority of reviewers have lessthan 200 reviews<jupyter_code>plt.figure(figsize=(12,5))
sns.distplot(counts, 100, kde=False, color='g')
plt.xlabel('Number of Movies Rated')
plt.ylabel('Proportion of Users')
plt.title('Distribution of User Behavior')
#plt.axis([10, 100])
plt.grid(True)
plt.show()
print('No. of Outliers ( users that have rated abnormally high number of movies): ',len(counts)-len(filtered))<jupyter_output>No. of Outliers ( users that have rated abnormally high number of movies):  9448<jupyter_text>Harder Threshold Filter reviewers with more than 100 and less than 10 reviews<jupyter_code>def hard_threshold(data):
    data = data[data < 100]
    return data[data > 10]
filtered_2 = hard_threshold(counts)
filtered_2.shape
plt.figure(figsize=(12,5))
# plt.hist(filtered_2, 50, density=False, facecolor='g', alpha=0.75)
sns.distplot(filtered_2, 50, kde=False, color='blue')
plt.xlabel('Number of Movies Rated')
plt.ylabel('Number of Users')
plt.title('Distribution of User Behavior - Filtered')
#plt.axis([10, 100])
plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>2. Distribution Pattern in Ratings<jupyter_code>y = data['rating'].value_counts()
y
ratings = y.iloc[:]
rcounts = y.index[:]
plt.figure(figsize=(6,5))
sns.distplot(data['rating'], 9, kde=False )
# plt.hist(data['rating'], 9, density=False, facecolor='b', alpha=0.75, histtype ='bar')
plt.xlabel('Rating Scale')
plt.ylabel('Number of Ratings')
plt.title('Distribution of User Ratings')
plt.show()
plt.figure(figsize=(6,5))
sns.boxplot(data['rating'])
plt.ylabel('Users')
plt.xlabel('Rating Scale')
plt.title('Distibution of Ratings')
plt.show()<jupyter_output><empty_output><jupyter_text>3. Number of Ratings collected over the Years<jupyter_code>date = list()
for i in tqdm(data['timestamp']):
    date.append(datetime.datetime.fromtimestamp(i).strftime('%Y'))
data['ratedYear'] = date
data.head()
z = data['ratedYear'].value_counts()
ycounts = z.iloc[:]
years = z.index[:]
df = pd.DataFrame({'Year':years,'counts':ycounts})
df.reset_index(drop =True)
df.sort_values(by=['Year'],inplace = True)
plt.figure(figsize=(12,5))
sns.barplot(df.Year, df.counts, palette='cool' )
plt.xticks(rotation = 90)
plt.xlabel('Year of the Rating')
plt.ylabel('Number of Ratings')
plt.title('Distribution of Rating Data over the Years')
plt.show()<jupyter_output><empty_output><jupyter_text>The last four years have seen an increase in ratings  4. Lifespan of Average Reviewer on the Movielens Data?<jupyter_code>df1 = data.groupby('ratedYear').rating.agg("mean")
df2 = data.groupby('userId').ratedYear.agg(["min","max"])
df2
xx= np.asarray(df2['max'].iloc[:])
yy= np.asarray(df2['min'].iloc[:])
value = list()
for i in range(len(df2)):
    v = int(df2['max'].iloc[i])-int(df2['min'].iloc[i])
    value.append(v)
lifespan = Counter(value)
lifespan.keys()
l1,l2 = [],[]

for i,j in lifespan.items():
    l1.append(i)
    l2.append(np.log(j))
l1
dff = pd.DataFrame({'Span':l1, 'Counts':l2})
dff.sort_values(by=['Span'], inplace = True)
dff.reset_index(drop=True, inplace = True)
dff.columns
plt.figure(figsize=(12,5))
# plt.bar(dff.Span, dff.Counts )
sns.barplot(dff.Span, dff.Counts, palette='terrain')
plt.xticks(rotation = 90)
plt.xlabel('Span of a User on the Platform ( in Years )')
plt.ylabel('Number of Users ( in Log Scale )')
plt.title('How long does a user use Movielens?')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.164985590778098, 'fraction_numerical': 0.021853986551392892, 'mean_word_length': 6.785046728971962, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '132614', 'license': 'BSD-4-Clause-UC', 'n_tokens_mistral': 1650, 'n_tokens_neox': 1523, 'n_words': 340, 'path': 'src/EDA.ipynb', 'repo_name': 'iopkelvin/Movie-Recommender', 'size': '4164'}