<jupyter_start><jupyter_text>This data set contains information about 10,000 movies collected from The Movie Database (TMDb), including user ratings and revenue.  Machine Learning Foundation Nanodegree  Project: Investigate a TMDb Movie Database Certain columns, like ‘cast’ and ‘genres’, contain multiple values separated by pipe (|) characters.There are some odd characters in the ‘cast’ column. Don’t worry about cleaning them. You can leave them as is.The final two columns ending with “_adj” show the budget and revenue of the associated movie in terms of 2010 dollars, accounting for inflation over time.  Introduction The dataset that I have chosen is the TMDb dataset which contains data on the movies and its ratings. The various trends that are to be explored in the same are going to be: - The popularity of the movies over the years- The revenue made by the movies over the years  We will see the comparison among the following:- Revenue versus Popularity - Revenue versus Votings  Let us see what are the questions that we will be answering:- What is the runtime of the movies, its revenue, and popularity over the years?- How do the votings affect the revenue of the movie?- Is revenue directly proportional to popularity? The dataset that we will be investigating in this project will be explanatory and any associations will be tentative as we will be just performing visualizations and basic correlations.  The three variables that we will be investigating are:- Single variable (1d) explorations that are Popularity, Revenue, Run-time over time - Multi-variable (2d) explorations that are Revenue over time for factors like Votings, Popularity<jupyter_code>import pandas as pd
import csv 
from datetime import datetime 
import matplotlib.pyplot as plt
% matplotlib inline<jupyter_output><empty_output><jupyter_text>Data Wrangling - Presenting the Data In this section of the report, we will load the data and check for cleanliness, and then trim and clean the dataset for analysis.<jupyter_code># Load your data and print out a few lines. Perform operations to inspect data
#   types and look for instances of missing or possibly errant data.

#loading the data form csv file for analyzing it 
df = pd.read_csv('tmdb-movies.csv', encoding='ISO-8859-1')
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 21 columns):
id                      10866 non-null int64
imdb_id                 10856 non-null object
popularity              10866 non-null float64
budget                  10866 non-null int64
revenue                 10866 non-null int64
original_title          10866 non-null object
cast                    10790 non-null object
homepage                2936 non-null object
director                10822 non-null object
tagline                 8042 non-null object
keywords                9373 non-null object
overview                10862 non-null object
runtime                 10866 non-null int64
genres                  10843 non-null object
production_companies    9836 non-null object
release_date            10866 non-null object
vote_count              10866 non-null int64
vote_average            10866 non-null float64
release_year            10866 non-null int64
budget_adj              1[...]<jupyter_text>Then we will be deleting all the duplicates that are present in the dataset<jupyter_code># check how many duplicated records are present
sum(df.duplicated())
# Delete the duplicated rows  

df.drop_duplicates(inplace=True)
#Check the dataset to make sure the duplicates are removed 

df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 10865 entries, 0 to 10865
Data columns (total 21 columns):
id                      10865 non-null int64
imdb_id                 10855 non-null object
popularity              10865 non-null float64
budget                  10865 non-null int64
revenue                 10865 non-null int64
original_title          10865 non-null object
cast                    10789 non-null object
homepage                2936 non-null object
director                10821 non-null object
tagline                 8041 non-null object
keywords                9372 non-null object
overview                10861 non-null object
runtime                 10865 non-null int64
genres                  10842 non-null object
production_companies    9835 non-null object
release_date            10865 non-null object
vote_count              10865 non-null int64
vote_average            10865 non-null float64
release_year            10865 non-null int64
budget_adj              1[...]<jupyter_text>NULL Values In this step we will look at the dataset and inspect if it is sufficicnt for our initial reserach. Then depending on the results, we will go furtehr into how are we going to deal with the missing values.<jupyter_code># To check if any of the columns have null values 

df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Data Cleaning  Now here, after we have seen the structure of our TMBd dataset and deleting the duplicated values, we will clean in this part. As we observed in the above step that few columns have null values. The only relevant thing for us is the imbd_id. We will be removing the null values in this section using dropna process. In the remaining columns that contain the missing values are not the factors that we need for our analysis.<jupyter_code># To drop the rows taht contain the missing values in imdb_id column

df.dropna(subset=['imdb_id'], inplace=True)  
df.info()
# To view the dataset 

df.head()
# Now we will save this new dataset which has no duplicates

df.to_csv('data_imdb.csv', index=False)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis  Question 1  What is the runtime of the movies, its revenue, and popularity over the years? In this analysis, we have to first group our dataset based on the years with the help of the variable 'release_year'.We will be performing the visualizations using in-built fucntions and histograms<jupyter_code># First we will import the data from the dataset that we cleaned above

df_imdb = pd.read_csv('data_imdb.csv')
df_imdb.info()
# Here we will group the input data by release_year variable and then compute the mean values for the numeric columns

df_1 = df_imdb.groupby('release_year').mean()
# To visualize the dataset 

df_1.hist(figsize=(15,15));<jupyter_output><empty_output><jupyter_text>Runtime  Here we will be determining if runtime has had any effect on the popularity of the movies. Then we will be visualizing it with the help of histograms.<jupyter_code>df_1['runtime'].hist(bins=30)
plt.xlabel('Runtime')
plt.ylabel('Counting')
plt.title('Runtime Over the Years');
# To view the visualization numerically

df_1['runtime'].describe()<jupyter_output><empty_output><jupyter_text>Observations - We can see that the runtime is popular in between 106 and 107 minutes - The movies fall in the range of 105 to 107 - The visualization shows that the distribution is right skewed   Popularity Here we want to see how the movie ratings will be deceived over the years. Visually the maximum count will reflect the popularity of the movie based on its ratings.<jupyter_code># Here we will look at the popularity variable in order to determine the ratings 

df_1['popularity'].hist(bins=38)
plt.xlabel('Popularity')
plt.ylabel('Countings')
plt.title('Popularity Over the Years');
# To view the visualization numerically

df_1['popularity'].describe()<jupyter_output><empty_output><jupyter_text>Observations - We can see that the maximum ratings fall in between 0.44 to 0.62- The disturbution is observed to be skewed to the right  Revenue Here we will determining the revenue of the movie that is generated over the years. Using histogram, we will be doing the visualizations and then the maximum counting will tell us about the revenue that is earned by the movies.<jupyter_code># Here we will look at the revenue variable over the years in order to determine typical revenues grossed by the movies

df_1['revenue'].hist(bins=26)
plt.xlabel('Revenue')
plt.ylabel('Countings')
plt.title('Revenue Over the Years');
# To view the visualization numerically

df_1['revenue'].describe()<jupyter_output><empty_output><jupyter_text>Observations We can observe that - The distribution is observed to be left skewed - The revenues are seen to be varying widely   Question 2 What are the variables which are associated with the revenue of the movies in the years and how?  The steps that we will be performing are:- In order to answer the above question, we will be looking at the correlation of the variables from the dataset that is grouped over the years- Here we will be visualizing the relation between two variables  The observations are based on:- Revenue vs Popularity- Revenue vs Votings<jupyter_code># Here we will use correlation to determine the factors which are influencing revenue 

df_1.corr(method='pearson')<jupyter_output><empty_output><jupyter_text>Revenue versus Popularity<jupyter_code># We will visualize this with the help of a scatterplot of revenue and popularity over the years

plt.scatter(x=df_1['revenue'], y=df_1['popularity'])
plt.xlabel('Revenue')
plt.ylabel('Popularity')
plt.title('Revenue vs Popularity Over the Years');<jupyter_output><empty_output><jupyter_text>Observations- Revenue vs Popularity In the above scatterplot, we can see that the popularity of the movies is directly proportional to the revenues.   Revenue versus Votings<jupyter_code># We will visualize this with the help of a scatterplot of revenue and votings over the years


plt.scatter(x=df_1['revenue'], y=df_1['vote_count'])
plt.xlabel('Revenue')
plt.ylabel('Votings')
plt.title('Revenue vs Votings Over the Years');<jupyter_output><empty_output><jupyter_text>Observations- Revenue vs Vote Count - The votings is almost proportional to the revenue of the movies - We can also see that few of the samples have some high voting counts and did not gross in the revenue section  Limitations - We worked with runtime of the movies, the popularity and revenue of the same - The analysis was restricted to the numerical values where dollar, rupees, etc was not considered. - If the varying currencies will be taken in consideration then the conversions or normalization needs to be performed- We have looked at other sources like the sales of the mvie tickets, the audio rights sales, sales of the broadcasts, etc. - The voting counts are taken at the face value where we did not look into the nature of the same whether it is good or bad. - The popularity medium is very much unknown which can have an impact on the analysis as limitations and the bias inherent are present in the end values.   Conclusion - From our EDA we analyzed that there are different patterns in the revenue, popularity and runtime of the movie over the years- The results are tentative but we observed the runtime to be popular in between 106 minutes and 107 minutes - The ratings were observed to be in the range of 0.44 to 0.62- We also found out that the Popularity of the movies and the revenue have a direct relation between them- The voting counts and the revenue is also observed to be related positively- The relations that we derived are just correlations<jupyter_code>from  subprocess  import call
call(['python', '-m', 'nbconvert', 'Investigate_a_Dataset.ipynb'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05183113364216415, 'fraction_numerical': 0.027532558342802202, 'mean_word_length': 3.9382822615451016, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '659362', 'license': 'MIT', 'n_tokens_mistral': 3227, 'n_tokens_neox': 2890, 'n_words': 1489, 'path': 'Investigate_a_Dataset.ipynb', 'repo_name': 'SparshaMishra/Investigate_a_Dataset', 'size': '11441'}