<jupyter_start><jupyter_text>MovieLens Data Exploration and Analysis  Perform Exploratory Data Analysis to identify the impact of various attributes on Movie Data Datasets Information:* **Data.csv:** It contains information of ratings given by the users to a particular movie.   - Columns: *user id, movie id, rating, timestamp** **item.csv:** File contains information related to the movies and its genre.  - Columns: *movie id, movie title, release date, unknown, Action, Adventure, Animation, Children’s, Comedy, Crime,       Documentary, Drama, Fantasy, Film-Noir, Horror, Musical, Mystery, Romance, Sci-Fi, Thriller, War, Western** **user.csv:** It contains information of the users who have rated the movies  - Columns: *user id, age, gender, occupation, zip code*   Learning Outcomes:* Exploratory Data Analysis* Visualization using Python* Pandas – groupby, merging Project - MovieLens Data AnalysisThe GroupLens Research Project is a research group in the Department of Computer Science and Engineering at the University of Minnesota. The data is widely used for collaborative filtering and other filtering solutions. However, we will be using this data to act as a means to demonstrate our skill in using Python to “play” with data. Domain Internet and Entertainment**Note that the project will need you to apply the concepts of groupby and merging extensively.**<jupyter_code>import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Import libraries 1. Import the necessary packages - 2.5 marks<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# To enable plotting graphs in Jupyter notebook
%matplotlib inline
import seaborn as sns
import datetime

from scipy.stats import zscore
from sklearn.preprocessing import StandardScaler, MinMaxScaler<jupyter_output><empty_output><jupyter_text>2. Read the 3 datasets into dataframes - 2.5 marks<jupyter_code># Load the data into pandas dataframe
urdf = pd.read_csv("Data.csv")
midf = pd.read_csv("item.csv")
uidf = pd.read_csv("user.csv")
urdf.shape
midf.shape
uidf.shape<jupyter_output><empty_output><jupyter_text>3. Apply info, shape, describe, and find the number of missing values in the data - 5 marks<jupyter_code>#Merge the User data frames using the 'user id' coloumn
merged_user_df = pd.merge(left=urdf, right=uidf, how='left', left_on='user id', right_on='user id')
merged_user_df.shape
merged_user_df.shape
merged_user_df.head()
#Merge the combined User data frame with the movie data frame using the 'movie id' coloumn
merged_movie_user_df = pd.merge(left=midf, right=merged_user_df, how='right', left_on='movie id', right_on='movie id')
merged_movie_user_df.shape
merged_movie_user_df.columns
merged_movie_user_df.head()
merged_movie_user_df.describe().T
# Check column types and missing values
merged_movie_user_df.info()
# From the information below we have 9 rows in the data frame with missing movie information (100000 - 99991 = 9)
def missing_check(df):
    total = df.isnull().sum().sort_values(ascending=False)   # total number of null values
    percent = (df.isnull().sum()/df.isnull().count()).sort_values(ascending=False)  # percentage of values that are null
    missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])  # putting the above two together
    return missing_data # return the dataframe
missing_check(merged_movie_user_df)
# Check missing values via heatmap
plt.figure(figsize=(8,6))
sns.heatmap(merged_movie_user_df.isna())
plt.show()<jupyter_output><empty_output><jupyter_text>4. Find the number of movies per genre using the item data - 5 marks<jupyter_code># use sum on the default axis
midf.sum()<jupyter_output><empty_output><jupyter_text>5. Find the movies that have more than one genre - 2.5 marks<jupyter_code>#hint: use sum on the axis = 1
temp = midf.drop(columns=['movie id', 'movie title', 'release date'])
out = temp.sum(axis=1)
index = 0
for i in out:
    if(i > 1):
        print(midf.loc[index,'movie title'],": ", i)
    index = index + 1<jupyter_output>Toy Story  :  3
GoldenEye  :  3
Get Shorty  :  3
Copycat  :  3
Twelve Monkeys  :  2
Babe  :  3
Richard III  :  2
Seven (Se7en)  :  2
Usual Suspects, The  :  2
Postino, Il  :  2
French Twist (Gazon maudit)  :  2
From Dusk Till Dawn  :  5
Angels and Insects  :  2
Muppet Treasure Island  :  5
Braveheart  :  3
Taxi Driver  :  2
Rumble in the Bronx  :  3
Apollo 13  :  3
Batman Forever  :  4
Crimson Tide  :  3
Desperado  :  3
Doom Generation, The  :  2
Free Willy 2: The Adventure Home  :  3
Mad Love  :  2
Net, The  :  2
Strange Days  :  3
Disclosure  :  2
<NAME>  :  2
Eat Drink Man Woman  :  2
Ed Wood  :  2
I.Q.  :  2
Star Wars  :  5
Legends of the Fall  :  4
Natural Born Killers  :  2
Outbreak  :  3
Professional, The  :  4
Pulp Fiction  :  2
Stargate  :  3
Santa Clause, The  :  2
What's Eating Gilbert Grape  :  2
While You Were Sleeping  :  2
Crow, The  :  3
Forrest Gump  :  3
Four Weddings and a Funeral  :  2
Lion King, The  :  3
Mask, The  :  3
Maverick  :  3
Faster Pussycat! K[...]<jupyter_text>6. Drop the movie where the genre is unknown - 2.5 marks<jupyter_code>#drop the empty NaN Values
merged_movie_user_df = pd.merge(left=midf, right=merged_user_df, how='left', left_on='movie id', right_on='movie id')

#drop the unknow Genre
merged_movie_user_df.drop(columns=['unknown'], inplace=True)<jupyter_output><empty_output><jupyter_text>7. Univariate plots of columns: 'rating', 'Age', 'release year', 'Gender' and 'Occupation' - 10 marks<jupyter_code># HINT: use distplot for age and countplot for gender,ratings,occupation.
# HINT: Please refer to the below snippet to understand how to get to release year from release date. You can use str.split()
# as depicted below
# Hint : Use displot without kde for release year or line plot showing year wise count.
a = 'My*cat*is*brown'
print(a.split('*')[3])

#similarly, the release year needs to be taken out from release date

#also you can simply slice existing string to get the desired data, if we want to take out the colour of the cat

print(a[10:])
print(a[-5:])

a = '01-Jan-1995'
print(a.split('-')[2])
#your answers here
sns.distplot(merged_movie_user_df['age']);
merged_movie_user_df['year'] = merged_movie_user_df['release date'].str.split('-').str.get(2)
sns.distplot(merged_movie_user_df['year'], kde=False);
#print(merged_movie_user_df['year'])
sns.countplot(merged_movie_user_df['gender']);
sns.countplot(merged_movie_user_df['rating']);
plt.figure(figsize=(24,16))
sns.countplot(merged_movie_user_df['occupation']);
plt.show()<jupyter_output><empty_output><jupyter_text>8. Visualize how popularity of genres has changed over the years - 10 marksNote that you need to use the number of releases in a year as a parameter of popularity of a genre Hint 1: you need to reach to a data frame where the release year is the index and the genre is the column names (one cell shows the number of release in a year in one genre) or vice versa.Once that is achieved, you can either use univariate plots or can use the heatmap to visualise all the changes over the years in one go. Hint 2: Use groupby on the relevant column and use sum() on the same to find out the nuumber of releases in a year/genre.<jupyter_code>#Your answer here
plt.figure(figsize=(40,24))

# Action Genere over years
ax = sns.barplot(x='year', y='Action', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))

# Adventure Genere over years
ax = sns.barplot(x='year', y='Adventure', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Animation Genere over years
ax = sns.barplot(x='year', y='Animation', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Childrens Genere over years
ax = sns.barplot(x='year', y='Childrens', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Comedy Genere over years
ax = sns.barplot(x='year', y='Comedy', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Crime Genere over years
ax = sns.barplot(x='year', y='Crime', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Drama Genere over years
ax = sns.barplot(x='year', y='Drama', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Fantasy Genere over years
ax = sns.barplot(x='year', y='Fantasy', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Film-Noir Genere over years
ax = sns.barplot(x='year', y='Film-Noir', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Horror Genere over years
ax = sns.barplot(x='year', y='Horror', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Musical Genere over years
ax = sns.barplot(x='year', y='Musical', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Mystery Genere over years
ax = sns.barplot(x='year', y='Mystery', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Romance Genere over years
ax = sns.barplot(x='year', y='Romance', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Sci-Fi Genere over years
ax = sns.barplot(x='year', y='Sci-Fi', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Thriller Genere over years
ax = sns.barplot(x='year', y='Thriller', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# War Genere over years
ax = sns.barplot(x='year', y='War', data=merged_movie_user_df)
plt.show()
plt.figure(figsize=(40,24))
# Western Genere over years
ax = sns.barplot(x='year', y='Western', data=merged_movie_user_df)
plt.show()<jupyter_output><empty_output><jupyter_text>9. Find the top 25 movies according to average ratings such that each movie has number of ratings more than 100 - 10 marksHint : 1. First find the movies that have more than 100 ratings(use merge, groupby and count). Extract the movie titles in a list.2. Find the average rating of all the movies and sort them in the descending order. You will have to use the .merge() function to reach to a data set through which you can get the names and the average rating.3. Use isin(list obtained from 1) to filter out the movies which have more than 100 ratings.Note: This question will need you to research about groupby and apply your findings. You can find more on groupby on https://realpython.com/pandas-groupby/.<jupyter_code>#your answer here
#Show the movies with their rating aggregate.
print('=================================================================')
print('')
df1 = merged_movie_user_df.groupby("movie title")["rating"].count().reset_index(name='rating')
print(df1)
#Show the movies that have a rating aggregate of more than 100.
print('')
print('=================================================================')
print('')
#df2 = merged_movie_user_df.groupby('movie title').size().reset_index(name='rating')
df2 = df1.loc[df1['rating'] > 100]
print(df2)
print('')
print('=================================================================')
print('')
#Show the movies with their averare aggregate rating in decending order.
df3 = merged_movie_user_df.groupby("movie title")["rating"].mean().reset_index(name='average')
print(df3.sort_values('average', ascending=False))
print('')
print('=================================================================')
print('')
#Show the movies that have aggregate rating more than 100 according to their average rating.
df3 = pd.merge(left=df2, right=df3, how='left', left_on='movie title', right_on='movie title')
#df3 = merged_movie_user_df.groupby("movie title")["rating"].mean().reset_index(name='average')
print(df3.sort_values('average', ascending=False))
print('')
print('=================================================================')<jupyter_output>=================================================================

                          movie title  rating
0                 'Til There Was You        9
1                              1-900        5
2                     101 Dalmatians      109
3                       12 Angry Men      125
4                                187       41
...                               ...     ...
1653                      Young Guns      101
1654                   Young Guns II       44
1655  Young Poisoner's Handbook, The       41
1656                Zeus and Roxanne        6
1657     Á köldum klaka (<NAME>)        1

[1658 rows x 2 columns]

=================================================================

                                 movie title  rating
2                            101 Dalmatians      109
3                              12 Angry Men      125
7                     2001: A Space Odyssey      259
15                           Absolute Power      127
16                     [...]<jupyter_text>10. See gender distribution across different genres check for the validity of the below statements - 10 marks* Men watch more drama than women* Women watch more Sci-Fi than men* Men watch more Romance than women 1. There is no need to conduct statistical tests around this. Just compare the percentages and comment on the validity of the above statements.2. you might want ot use the .sum(), .div() function here.3. Use number of ratings to validate the numbers. For example, if out of 4000 ratings received by women, 3000 are for drama, we will assume that 75% of the women watch drama.  Conclusion:<jupyter_code>#Men watch more drama than women
print('=================================================================')
print('')
drama = merged_movie_user_df.groupby(by=["gender", "Drama"])["rating"].count().reset_index(name='rating')
print(drama)
print('')
# Men and women Watch the same percentage of Drama. This statement is false.
#  Men % 0.5 =  28887 / (45366 + 28887)
# Women % 0.5 = 11008 / (11008 + 14730)

#Women watch more Sci-Fi than men
print('=================================================================')
print('')
sci_fi = merged_movie_user_df.groupby(by=["gender", "Sci-Fi"])["rating"].count().reset_index(name='rating')
print(sci_fi)
print('')
# Men watch more Si-Fi than Woment. This statement is false.
#  Men % 0.136 =  10101 / (64152 + 10101)
# Women % 0.102 = 2629 / (23109 + 2629)


#Men watch more Romance than women
print('=================================================================')
print('')
Romance = merged_movie_user_df.groupby(by=["gender", "Romance"])["rating"].count().reset_index(name='rating')
print(Romance)
print('')
# Men watch more romance than Women. This statement is True.
#  Men % 0.389 =  28887 / (28887 + 45366)
# Women % 0.227 = 5858 / (5858 + 19880)
print('=================================================================')<jupyter_output>=================================================================

  gender  Drama  rating
0      F      0   14730
1      F      1   11008
2      M      0   45366
3      M      1   28887

=================================================================

  gender  Sci-Fi  rating
0      F       0   23109
1      F       1    2629
2      M       0   64152
3      M       1   10101

=================================================================

  gender  Romance  rating
0      F        0   19880
1      F        1    5858
2      M        0   60650
3      M        1   13603

=================================================================
{'fraction_non_alphanumeric': 0.16557377049180327, 'fraction_numerical': 0.033836065573770495, 'mean_word_length': 4.338116905845292, 'pattern_counts': {'":': 1, '<': 44, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '647461', 'license': 'MIT', 'n_tokens_mistral': 5174, 'n_tokens_neox': 4627, 'n_words': 1599, 'path': 'project1/Shashi_proj1_MovieLens.ipynb', 'repo_name': 'shashidande-PGP-AIML/AIML-Projects', 'size': '15265'}