<jupyter_start><jupyter_text>Project - EDA of IMDB dataset  Index[1. Introduction](intro) [2. Data Wrangling](wrangle) [3. EDA (exploratory data analysis)](eda)[4. Conclusions](con)   1. Introduction<jupyter_code>import pandas as pd
import numpy as np
from pprint import pprint
import matplotlib.pyplot as plt
import seaborn as sns
import math
%matplotlib inline
df = pd.read_csv('tmdb_movies.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Summary of the dataset:     The dataset chosen for EDA is the imdb data set with 21 columns. It provides data of movies released and other features like release date, cast, director, budget, revenue , populairty etc. Questions that can be explored from the dataset are :Which genres are most popular from year to year? What kinds of properties are associated with movies that have high revenues?  Questions to be answered1.  ResourcesDAND program material and other online material from websites like stackoverflow and documentation of libraries   2. Data Wrangling Data is loaded, is provided by imdb and can be downloaded [here](https://docs.google.com/document/d/e/2PACX-1vTlVmknRRnfy_4eTrjw5hYGaiQim5ctr9naaRd4V9du2B5bxpd8FEH3KtDgp8qVekw7Cj1GLk1IXdZi/pub?embedded=True). It is a good idea to explore the data using basic functions like : (describe, len, shape, anynull etc), look at the head tail, len etc and get an idea of what we are working with. There are certian columns which we will omit to make it easy for us to explore like is, overview, production companies etc)  Steps taken to clean the dataset1. Changed release date to datetime data-type2. Checked for null-values and removed the columns with highest null values which are not used in the analysis - ('overview', 'keywords', 'production_companies', 'id', 'homepage','tagline')3. Removed the rows with null values using 'dropna'4. the original shape of the dateset (10866, 21) was changed to (10725, 16)<jupyter_code>df.head()
df.shape
print('The shape of the imdb dataset is {}\n'.format(df.shape))
print('The basic describe function returns the following\n')
df.describe()

df.info()

df['release_date'] = pd.to_datetime(df['release_date'])
'''
release_date is converted to datetime to make analysis easy using pandas
'''
display(df.isnull().any())
'''
checking for null values
'''
print('The no.of columns with null values are {}\n\n'.format(df.isnull().any().sum()))
display(df.isnull().sum())
print('The total no.of null values in the dataset are {}\n\n'.format(df.isnull().sum().sum()))

display(df.isnull().sum())<jupyter_output><empty_output><jupyter_text>Dropping Null valuesHomepage, tagline, keywords, production_companies have many null values and are not used in the analysis hence are dropped. Having two id is useless in the analysis so and imdb id is taken as unique id. Removing all null values without changing the shape of the data set considerably.<jupyter_code>df.drop(['overview', 'keywords', 'production_companies', 'id', 'homepage','tagline' ], axis=1, inplace = True)
df.isnull().sum().sum()
print('The no of null values after dropping unused columns {}\n'.format(df.isnull().sum().sum()))
df.dropna(axis = 0, how = 'any', inplace = True)
'''
Dropping rows with any null values
'''
final = df.shape
print('The final shape of the dataset is {}\nAnd the no of null values is {}\n'.format(df.shape, df.isnull().sum().sum()))


df.hist(figsize = (17,12), log = True);
'''
log tranforamtion is used to make the plot more easy to read
'''<jupyter_output><empty_output><jupyter_text>From the histogram above certain trends can be seen 1. Budget - It is right skewed, that is more no of movies have budget less then 1*le8. and the max is at 4.5*le8 (approx).2. Popularity - It is right skewed with feew outliers, which means certain movies have received outstanding "popularity" than their peers.3. Release_year - It is left skewed, and the no of movies released increased from 1960 to 1990(slow pace)  and from 1990 to 2010(avg pace)4. Revenue - It is right skewed, and majority of the movies have grossed under 1*le95. Runtime - Majority of the movies have runtime under 200 minutes6. Voter_average - It looks like normal distribution, with majority at 6.<jupyter_code>'''
using corr to find the co-relation between variables and plotting them using seaborn and matplotlib
'''
f, ax = plt.subplots(figsize=(10, 8));
corr = df.corr()
sns.heatmap(corr, mask=np.zeros_like(corr, dtype=np.bool), cmap=sns.diverging_palette(220, 10, as_cmap=True),
            square=True, ax=ax);<jupyter_output><empty_output><jupyter_text>1. Co-relation appears strong between budget and (popularity, revenue, voter_count)<jupyter_code># save new clean dataset which contains no duplicates or records with missing values in of interest.
# we will use this dataset in next sections
df.to_csv('clean_imdb.csv', index=False)<jupyter_output><empty_output><jupyter_text>3. EDA"""one dependent variable and three independent variables """ """single variable 1d and multi variable 2d""""""Reasoning is provided for each analysis decision, plot, and statistical summary.""""""Use statistical tests to draw rigorous conclusions where appropriate."""<jupyter_code>df = pd.read_csv('clean_imdb.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Question 1How are Runtime, popularity, revunue changing overtime and is there a interrelatioon between them?  Runtime     Over the years runtime has relatively remained the same, the hasnot been major shift, but after 1990 there have been ouliers or movies whose runtime is outside the 200 minute mark. This can be seen in the scatterplot       As seen from the description 50% of movies run between 90 to 112 minutes.    Runtime and popularity is  normal distribution  if outliers are removed and a transformation is applied to smoothen the curve. And around 80% of the runtime is between<jupyter_code>df['runtime'].describe() 
x= df['release_year']
y= df['runtime']
plt.scatter(x, y, c="g")
plt.xlabel("release year")
plt.ylabel("Runtime")
plt.show()<jupyter_output><empty_output><jupyter_text>Popularity     There is a direct relationship between popularity and vote average, this can be attributed to the likeliness of more people voting on a popular movie than a unpopular one.    Over the years popularity of movies has grown and this is because more people owning televison in the 90s and the boom of  multiplex and television in the 2000s and later period.    Revenue and budget do not have major impact on populairty<jupyter_code>df['popularity'].describe() 
# create a figure
fig = plt.figure(figsize=(10, 10))
 
# define subplots and their positions in figure
plt1 = fig.add_subplot(221)
plt2 = fig.add_subplot(222)
plt3 = fig.add_subplot(223)
plt4 = fig.add_subplot(224)
 
# plotting points on each subplot
plt1.scatter(df['revenue'], df['popularity'], color ='r')
plt1.set_title('Revenue vs Popularity')
plt1.set_xlabel('revenue')
plt1.set_ylabel('popularity')

plt2.scatter(df['budget'], df['popularity'], color ='b')
plt2.set_title('Budget vs Popularity')
plt2.set_xlabel('Budget')
plt2.set_ylabel('popularity')
 
plt3.scatter(df['vote_average'], df['popularity'], color ='g')
plt3.set_title('Vote average vs Popularity')
plt3.set_xlabel('Vote Average')
plt3.set_ylabel('popularity')

 
plt4.scatter(df['release_year'], df['popularity'], color ='k')
plt4.set_title('release_year vs popularity')
plt4.set_xlabel('Release Year')
plt4.set_ylabel('popularity')

plt.show()



df['runtime'].describe() 
plt.scatter(df['runtime'], df['popularity'], color ='k')
plt.title('release_year vs popularity')
plt.xlabel('Runtime(min)')
plt.ylabel('popularity')<jupyter_output><empty_output><jupyter_text>Revunue    Overtime the revenue of movies have increased. In 1990 there is major shift in popularity, another one is at 2000. Since 1990 the movies became a profitable business.<jupyter_code>df['revenue'].describe() 
x= df['release_year']
y= df['popularity']
p= df['revenue']-df['budget']

plt.scatter(x, y)
plt.xlabel("release year")
plt.ylabel("popularity")
plt.show()
fig, ax = plt.subplots(figsize=(20,7))
p= df['revenue']-df['budget']
scat = ax.scatter(df['release_year'],df['revenue'],  c=p, s=200, marker='o')
fig.colorbar(scat)

plt.show()<jupyter_output><empty_output><jupyter_text>Question 2 :    Which is the most famous genre?    The genres are spilt and made into an array. Finally the counts of individual genre is found to give the famous genre.<jupyter_code>df2 = df['genres'].str.split('|', expand = True)
df2
df3 = df2.values
df3
df5 = df3[df3 != np.array(None)]
df6 = pd.DataFrame(df5)
df6[0].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1098225675518961, 'fraction_numerical': 0.02133828134060072, 'mean_word_length': 5.669760247486465, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '623555', 'license': 'MIT', 'n_tokens_mistral': 2843, 'n_tokens_neox': 2626, 'n_words': 977, 'path': 'final_code.ipynb', 'repo_name': 'chrisdmell/IMDB_dataset', 'size': '8623'}