<jupyter_start><jupyter_text>IMDB Dataset Analysis<jupyter_code># Supress Warnings

import warnings
warnings.filterwarnings('ignore')
# Import the numpy and pandas packages

import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Task 1: Reading and Inspection-   Subtask 1.1: Import and readImport and read the movie database. Store it in a variable called `movies`.<jupyter_code># Write your code for importing the csv file here
movies = pd.read_csv('Movie+Assignment+Data.csv')
movies.head()
# Last 5 rows
movies.tail()<jupyter_output><empty_output><jupyter_text>-   Subtask 1.2: Inspect the dataframeInspect the dataframe's columns, shapes, variable types etc.<jupyter_code># Write your code for inspection here
movies.info()
movies.describe().T<jupyter_output><empty_output><jupyter_text>Some key point from above table- Avg movie duration is 107.2 minuts- Avg imdb is 6.44- Avg number of users reviews is 272<jupyter_code># Write your code for inspection here
a = movies.shape
a

movies.columns

movies.index<jupyter_output><empty_output><jupyter_text>Task 2: Cleaning the Data-   Subtask 2.1: Inspect Null valuesFind out the number of Null values in all the columns and rows. Also, find the percentage of Null values in each column. Round off the percentages upto two decimal places.<jupyter_code># Write your code for column-wise null count here
movies.isnull().sum()                                                 
# Write your code for row-wise null count here
movies.isnull().sum(axis = 1)                                  
# Write your code for column-wise null percentages here

(movies.isnull().mean()*100).round(2)<jupyter_output><empty_output><jupyter_text>-   Subtask 2.2: Drop unecessary columnsFor this assignment, you will mostly be analyzing the movies with respect to the ratings, gross collection, popularity of movies, etc. So many of the columns in this dataframe are not required. So it is advised to drop the following columns.-  color-  director_facebook_likes-  actor_1_facebook_likes-  actor_2_facebook_likes-  actor_3_facebook_likes-  actor_2_name-  cast_total_facebook_likes-  actor_3_name-  duration-  facenumber_in_poster-  content_rating-  country-  movie_imdb_link-  aspect_ratio-  plot_keywords<jupyter_code># Write your code for dropping the columns here. It is advised to keep inspecting the dataframe 
#after each set of operations 

to_drop = ['color', 'director_facebook_likes','actor_1_facebook_likes','actor_2_facebook_likes',
           'actor_3_facebook_likes', 'actor_2_name','cast_total_facebook_likes','actor_3_name',
           'duration','facenumber_in_poster','content_rating','country','movie_imdb_link','aspect_ratio',
           'plot_keywords']

# Taking all column names to list
my_col = movies.columns.to_list()

new_col = []
[new_col.append(i) for i in my_col if i not in to_drop]

print(new_col)             # These columns are to be retained

movies = movies[['movie_title','actor_1_name','director_name',  'language','genres','title_year','budget', 'gross', 'imdb_score','num_critic_for_reviews', 'num_voted_users',  'num_user_for_reviews','movie_facebook_likes']]
movies.head()    # Reshuffling DataFrame
#Checking Shape after removing columns
movies.shape       
movies.describe().T<jupyter_output><empty_output><jupyter_text>-   Subtask 2.3: Drop unecessary rows using columns with high Null percentagesNow, on inspection you might notice that some columns have large percentage (greater than 5%) of Null values. Drop all the rows which have Null values for such columns.<jupyter_code># Columnwise null values

movies.isnull().mean(axis = 0)*100<jupyter_output><empty_output><jupyter_text>We can see 'gross' and 'budget' columns have more than 5% of null values<jupyter_code># Write your code for dropping the rows here
movies.dropna(axis = 0, subset = ['gross', 'budget'], inplace = True)
movies.head()
#Checking Shape
movies.shape
#Checking % of Null Values in columns
movies.isnull().mean(axis = 0)*100<jupyter_output><empty_output><jupyter_text>-   Subtask 2.4: Fill NaN valuesYou might notice that the `language` column has some NaN values. Here, on inspection, you will see that it is safe to replace all the missing values with `'English'`.<jupyter_code>movies['language'].isnull().sum()
movies[movies['language'].isnull()== True]
# Write your code for filling the NaN values in the 'language' column here
movies['language'].fillna(value = 'English', inplace = True)
#Checking 
movies.loc[[3086, 4110, 4958], :]
#Checking % of Null Values in columns
movies.isnull().mean(axis = 0)*100<jupyter_output><empty_output><jupyter_text>-   Subtask 2.5: Check the number of retained rowsYou might notice that two of the columns viz. `num_critic_for_reviews` and `actor_1_name` have small percentages of NaN values left. You can let these columns as it is for now. Check the number and percentage of the rows retained after completing all the tasks above.<jupyter_code>movies.shape<jupyter_output><empty_output><jupyter_text>Total rows retained = 3891<jupyter_code>b = movies.shape
b
# No. of Rows in initial dataset
a[0] 
# No of Rows after Cleaning Dataset
b[0]
# Write your code for checking number of retained rows here
retained_rows_percent  = b[0]*100/a[0]
retained_rows_percent<jupyter_output><empty_output><jupyter_text>Rows retained = 77.16% **Checkpoint 1:** You might have noticed that we still have around `77%` of the rows!  Task 3: Data Analysis-   Subtask 3.1: Change the unit of columnsConvert the unit of the `budget` and `gross` columns from `$` to `million $`.<jupyter_code>movies.head()

# Write your code for unit conversion here from $ to Million $
movies[['budget', 'gross']] = movies[['budget', 'gross']].apply(lambda x: x/1000000)
movies.head()
movies.describe().T<jupyter_output><empty_output><jupyter_text>-   Subtask 3.2: Find the movies with highest profit    1. Create a new column called `profit` which contains the difference of the two columns: `gross` and `budget`.    2. Sort the dataframe using the `profit` column as reference.    3. Plot `profit` (y-axis) vs `budget` (x- axis) and observe the outliers using the appropriate chart type.    4. Extract the top ten profiting movies in descending order and store them in a new dataframe - `top10`<jupyter_code># Write your code for creating the profit column here
movies['profit'] = movies['gross'] - movies['budget']         
movies.head()
movies.shape
# Write your code for sorting the dataframe here
movies.sort_values(by = 'profit', ascending = False, inplace = True)
movies.head(10)
movies.tail()
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
sns.set_style('whitegrid')

# Write code for profit vs budget plot here
plt.figure(figsize = (10,8))
plt.scatter(x= movies['budget'], y = movies['profit'], c = 'r', )
plt.xlabel("Budget")
plt.ylabel("Profit")
plt.title("Budget Vs Profit")
plt.show()
#Plot after removing outliers

plt.figure(figsize = (10,8))
sns.jointplot(x= 'budget', y = 'profit', data = movies.iloc[:-14, :], kind = 'scatter')

plt.show()
# Write your code to get the top 10 profiting movies here

top10 = movies.iloc[:10, :]
top10<jupyter_output><empty_output><jupyter_text>Here we can Observe The Avengers movie is repeated so it means we have duplicates -   Subtask 3.3: Drop duplicate valuesAfter you found out the top 10 profiting movies, you might have noticed a duplicate value. So, it seems like the dataframe has duplicate values as well. Drop the duplicate values from the dataframe and repeat `Subtask 3.2`. Note that the same `movie_title` can be there in different languages.<jupyter_code>#Checking - searching Duplicate Rows based on movie_title and language
pd.set_option('display.max_rows', 200)

dupli_bool = movies.duplicated(subset = 'movie_title', keep = False)   
duplicate = movies.loc[dupli_bool == True]

print(duplicate[['movie_title', 'language']])<jupyter_output>movie_title  language
794                                The Avengers    English
17                                 The Avengers    English
1805                            The Jungle Book    English
79                              The Jungle Book    English
187   The Twilight Saga: Breaking Dawn - Part 2    English
174   The Twilight Saga: Breaking Dawn - Part 2    English
1378                                   Twilight    English
2262                                   Twilight    English
3463                                       Juno    English
4557                                       Juno    English
33                          Alice in Wonderland    English
4128                        Alice in Wonderland    English
100                    The Fast and the Furious    English
395                    The Fast and the Furious    English
1332                   The Fast and the Furious    English
336                                  Cinderella    Engli[...]<jupyter_text>We can See the movie 'The Host' is in Korean and English<jupyter_code>movies.shape
# Write your code for dropping duplicate values here
movies.drop_duplicates(subset = None, inplace = True)
movies.head(10)
movies.shape
# Checking -  35 rows are removed

pd.set_option('display.max_rows', 200)

dupli_bool2 = movies.duplicated(subset = 'movie_title', keep = False)
duplicate2 = movies.loc[dupli_bool2 == True]

duplicate2<jupyter_output><empty_output><jupyter_text>We can still see similar rows but they are different in mostly one column "num_voted_users "<jupyter_code># Checking 'The Host' is not removed from the dataframe
movies.loc[[1002, 2988], :]
# Repeat subtask 3.2
#     Subtask 3.2: Find the movies with highest profit
#         Create a new column called profit which contains the difference of the two columns: gross and budget.
#         Sort the dataframe using the profit column as reference.
#         Plot profit (y-axis) vs budget (x- axis) and observe the outliers using the appropriate chart type.
#         Extract the top ten profiting movies in descending order and store them in a new dataframe - top10<jupyter_output><empty_output><jupyter_text>movies dataframe already having profit column and sorted* Create a new column called profit which contains the difference of the two columns: gross and budget.* Sort the dataframe using the profit column as reference.<jupyter_code>movies.head()
movies.shape
# Write code for repeating subtask 2 here
# Write code for profit vs budget plot here
# Write code for profit vs budget plot here
plt.figure(figsize = (10,8))
plt.scatter(x= movies['budget'], y = movies['profit'], c = 'g', )
plt.xlabel("Budget")
plt.ylabel("Profit")
plt.title("Budget Vs Profit")
plt.show()
# Write code for repeating subtask 2 here
# Write your code to get the top 10 profiting movies here

top10 = movies.iloc[:10, :]

top10<jupyter_output><empty_output><jupyter_text>We can see the duplicate row removed , Two movies directed by <NAME> in the list are Avatar and Titanic **Checkpoint 2:** You might spot two movies directed by `<NAME>` in the list. -   Subtask 3.4: Find IMDb Top 250    1. Create a new dataframe `IMDb_Top_250` and store the top 250 movies with the highest IMDb Rating (corresponding to the column: `imdb_score`). Also make sure that for all of these movies, the `num_voted_users` is greater than 25,000.Also add a `Rank` column containing the values 1 to 250 indicating the ranks of the corresponding films.    2. Extract all the movies in the `IMDb_Top_250` dataframe which are not in the English language and store them in a new dataframe named `Top_Foreign_Lang_Film`.<jupyter_code># Creating a copy of movies dataframe - This dataframe is cleaned of NaN and Duplicate rows

IMDB_movies = movies.copy()
IMDB_movies.head()
IMDB_movies.shape
IMDB_movies.columns
# Since there are several similar rows with different 'num_voted_users' we take mean of that column 

IMDB_movies = IMDB_movies.groupby(['movie_title', 'actor_1_name', 'director_name', 'language', 'genres',
       'title_year', 'budget', 'gross', 'imdb_score', 'num_critic_for_reviews',
        'num_user_for_reviews', 'movie_facebook_likes',
       'profit'])['num_voted_users'].mean()
IMDB_movies = pd.DataFrame(IMDB_movies)
IMDB_movies.reset_index(inplace = True)
IMDB_movies.shape
IMDB_movies.head()
# Checking duplicate rows having same movie_title

pd.set_option('display.max_rows', 200)

dupli_bool3 = IMDB_movies.duplicated(subset = 'movie_title', keep = False)
duplicate3 = IMDB_movies.loc[dupli_bool3 == True]

duplicate3
# Write your code for extracting the top 250 movies as per the IMDb score here. 
#Make sure that you store it in a new dataframe and name that dataframe as 'IMDb_Top_250'

# First sorting by IMDb score

IMDB_movies.sort_values(by = 'imdb_score', ascending = False, inplace = True)   # sort by imdb score
IMDB_movies.head()
#Filtering based on 'num_voted_users' > 25000

IMDB_movies = IMDB_movies[IMDB_movies['num_voted_users'] > 25000]
IMDB_movies.head()
IMDB_movies.shape<jupyter_output><empty_output><jupyter_text>So now only 2544 rows are remaining<jupyter_code># Top 250 movies based on imdb score and Filtering based on 'num_voted_users' > 25000

IMDb_Top_250 = IMDB_movies.iloc[:250, :]               
IMDb_Top_250.head()
IMDb_Top_250.shape
# Making a new rank column based on IMDb score

IMDb_Top_250['rank'] = IMDb_Top_250['imdb_score'].rank(ascending = False, method = 'first')
IMDb_Top_250<jupyter_output><empty_output><jupyter_text>Veer Zara Rank is 238<jupyter_code>#Extract all the movies in the IMDb_Top_250 dataframe which are not in the English language and 
#store them in a new dataframe named Top_Foreign_Lang_Film.
# Write your code to extract top foreign language films from 'IMDb_Top_250' here

Top_Foreign_Lang_Film = IMDb_Top_250[IMDb_Top_250['language'] != 'English']
Top_Foreign_Lang_Film<jupyter_output><empty_output><jupyter_text>Veer Zaara is fourth last **Checkpoint 3:** Can you spot `Veer-Zaara` in the dataframe? -  Subtask 3.5: Find the best directors    1. Group the dataframe using the `director_name` column.    2. Find out the top 10 directors for whom the mean of `imdb_score` is the highest and store them in a new dataframe `top10director`.  Incase of a tie in IMDb score between two directors, sort them alphabetically.<jupyter_code>#Making a working dataframe

Director_movies = movies.copy()
Director_movies.shape
# Write your code for extracting the top 10 directors here

top10director = Director_movies.groupby( ['director_name'])['imdb_score'].mean().sort_values(ascending = False)
top10director = pd.DataFrame(top10director)

top10director = top10director.sort_values(by= ['imdb_score', 'director_name'], ascending= [False, True])
top10director = top10director.iloc[:10, :]
top10director<jupyter_output><empty_output><jupyter_text>**Checkpoint 4:** No surprises that `<NAME>` (director of Whiplash and La La Land) is in this list. -   Subtask 3.6: Find popular genresYou might have noticed the `genres` column in the dataframe with all the genres of the movies seperated by a pipe (`|`). Out of all the movie genres, the first two are most significant for any film.1. Extract the first two genres from the `genres` column and store them in two new columns: `genre_1` and `genre_2`. Some of the movies might have only one genre. In such cases, extract the single genre into both the columns, i.e. for such movies the `genre_2` will be the same as `genre_1`.2. Group the dataframe using `genre_1` as the primary column and `genre_2` as the secondary column.3. Find out the 5 most popular combo of genres by finding the mean of the gross values using the `gross` column and store them in a new dataframe named `PopGenre`.<jupyter_code># Making Working dataframe

Popular_genre = movies.copy()
Popular_genre.head()
Popular_genre.tail(10)<jupyter_output><empty_output><jupyter_text>We can see row 3075 etc has only one genre<jupyter_code># Write your code for extracting the first two genres of each movie here

Popular_genre['genre_1'] = Popular_genre['genres'].str.split('|').str[0]

Popular_genre['genre_2'] = Popular_genre['genres'].str.split('|').str[1]

Popular_genre.head()

Popular_genre.tail(10)<jupyter_output><empty_output><jupyter_text>Column genre_2 , row 3075 is filled with NaN<jupyter_code>#Replacing NaN with genre_1 values

Popular_genre['genre_2'].fillna(Popular_genre['genres'].str.split('|').str[0], inplace = True)
Popular_genre.tail(10)
Popular_genre.info()
# Write your code for grouping the dataframe here

movies_by_segment = Popular_genre.groupby(['genre_1', 'genre_2']).mean()
movies_by_segment.head()
# Write your code for getting the 5 most popular combo of genres here
PopGenre = Popular_genre.groupby(['genre_1', 'genre_2'])['gross'].mean().sort_values(ascending = False)
PopGenre = PopGenre[:5]

PopGenre = pd.DataFrame(PopGenre)
PopGenre<jupyter_output><empty_output><jupyter_text>**Checkpoint 5:** Well, as it turns out. `Family + Sci-Fi` is the most popular combo of genres out there! -   Subtask 3.7: Find the critic-favorite and audience-favorite actors    1. Create three new dataframes namely, `Meryl_Streep`, `Leo_Caprio`, and `Brad_Pitt` which contain the movies in which the actors: '<NAME>', '<NAME>', and '<NAME>' are the lead actors. Use only the `actor_1_name` column for extraction. Also, make sure that you use the names '<NAME>', '<NAME>', and '<NAME>' for the said extraction.    2. Append the rows of all these dataframes and store them in a new dataframe named `Combined`.    3. Group the combined dataframe using the `actor_1_name` column.    4. Find the mean of the `num_critic_for_reviews` and `num_users_for_review` and identify the actors which have the highest mean.    5. Observe the change in number of voted users over decades using a bar chart. Create a column called `decade` which represents the decade to which every movie belongs to. For example, the  `title_year`  year 1923, 1925 should be stored as 1920s. Sort the dataframe based on the column `decade`, group it by `decade` and find the sum of users voted in each decade. Store this in a new data frame called `df_by_decade`.<jupyter_code># Create working Dataframe

Favourite_actor = movies.copy()
Favourite_actor.head()
# Write your code for creating three new dataframes here

# Include all movies in which Meryl_Streep is the lead

Meryl_Streep = Favourite_actor[Favourite_actor['actor_1_name'] == '<NAME>']
Meryl_Streep.head()                                                                 #Expected Output
Meryl_Streep.shape
Leo_Caprio = Favourite_actor[Favourite_actor['actor_1_name'] == '<NAME>']
Leo_Caprio.head()                                                                         #Expected Output  
Leo_Caprio.shape
Brad_Pitt = Favourite_actor[Favourite_actor['actor_1_name'] == '<NAME>']
Brad_Pitt.head()                                                                        #Expected Output 
Brad_Pitt.shape
# Write your code for combining the three dataframes here

Combined = pd.concat([Meryl_Streep, Leo_Caprio, Brad_Pitt])
Combined.head()                                                     #Expected Output

Combined.shape
# Write your code for grouping the combined dataframe here

Combined.groupby(['actor_1_name']).mean()                                   #Expected Output

# Write the code for finding the mean of critic reviews  here
Critic_Review = Combined.groupby(['actor_1_name'])['num_critic_for_reviews'].mean().sort_values(ascending = False)
Critic_Review                                    #Expected Output

# Write the code for finding the mean of  audience reviews here
User_Review = Combined.groupby(['actor_1_name'])['num_user_for_reviews'].mean().sort_values(ascending = False)
User_Review                                     #Expected Output<jupyter_output><empty_output><jupyter_text>num_critic_for_reviews  =  Leonardo DiCaprio =    330.190476 num_user_for_reviews    =  Leonardo DiCaprio =    914.476190 **Checkpoint 6:** `Leonardo` has aced both the lists!<jupyter_code># Creating working Dataframe

Decade = movies.copy()
Decade.head()
# Write the code for calculating decade here
Decade['decade'] = ((Decade['title_year']//10) * 10)
Decade.head()                                                     #Expected Output
Decade.sort_values(by = 'decade', inplace = True)
Decade.head() 
# Write your code for creating the data frame df_by_decade here 
df_by_decade = Decade.groupby('decade')['num_voted_users'].sum()
df_by_decade = pd.DataFrame(df_by_decade)
df_by_decade.reset_index(inplace = True)
df_by_decade                                            #Expected Output

# Write your code for plotting number of voted users vs decade
plt.figure(figsize = (10, 8))
sns.barplot(x = 'decade', y = 'num_voted_users' , data = df_by_decade)
plt.xlabel("Decade")
plt.ylabel('User_Votes')
plt.yscale('log')
plt.title("BARPLOT - User Votes vs Decade")
plt.show()                                                          #Expected Output<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08917988912253871, 'fraction_numerical': 0.02141082011087746, 'mean_word_length': 4.098684210526316, 'pattern_counts': {'":': 0, '<': 116, '<?xml version=': 0, '>': 119, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 30, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '65343', 'license': 'MIT', 'n_tokens_mistral': 6805, 'n_tokens_neox': 6350, 'n_words': 2208, 'path': 'IMDB_Data_Analysis.ipynb', 'repo_name': 'sudevansujit/IMDB_Data_Analysis', 'size': '21007'}