<jupyter_start><jupyter_text>![alt text](download.png)>  Student: <NAME> >  Date: April 9, 2019>  Dataset: The Movie Database TMDb     Project: Investigate the Top Rating Films Contents>> Introduction> Data Wrangling> Exploratory Data Analysis> Conclusions>  Introduction> In this project, I will be analyzing movies data provided by Kaggle. The given TMDb dataset contains 10866 entries and 21 columns, various information about movies. It covers time duration from 1960 to 2015 (55 years). The information includes quantative data as budget, revenue combined with cathegorical data as names of director, actors and genres. >> My goal is to gain insight into dataset structure, to identify influential feature and to find the most successful movies during the 55-year time period. The success factors would include profitability and popularity. To valuate movies earning power, a net profit margin ratio will be calculated. The margin ratio (percentage) would reveal the remaining profit after all costs have been deducted. In my view, it is an accurate metric of profitability. >> In the data set the last two columns ending with 'adj' (budget_adj, revenue_adj) show the budget and revenue of the associated movie in terms of 2010 dollars, accounting for inflation over time. According to the Bureau of Labor Statistics consumer price index, prices in 2015 are 700.73% higher than average prices throughout 1960. This is suggesting that inflation rate is significaly high, and that's why I will only use data in columns ending with 'adj' disregarding values in columns 'revenue' and 'budget'.  Data WranglingAssesing and Building Intuition>Examine the shape of data set, number of rows and columns >Examine meaning of each feature>Examine and convert datatypes>Examine and remove duplicates, null values, missing values, extraneous columns<jupyter_code># importing libraries
import pandas as pd
import scipy as sp
import seaborn as sn
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
# loading movies data into data frame
df = pd.read_csv('tmdb-movies.csv')

# showing only one row of data
df.head(1)
# checking how big is dataset
df.shape<jupyter_output><empty_output><jupyter_text>>10 866 movies (rows) and 21 feature (column) in total<jupyter_code># checking how many unique data points in every columns
df.nunique()<jupyter_output><empty_output><jupyter_text>>The number less than 10865 indicates duplicates in the column. As seen above, all columns, except 'id'column, have duplicates.<jupyter_code># droping duplicates
df.drop_duplicates(inplace=True)
#filling in missing values with mean
df.fillna(df.mean(), inplace=True)
#deleting null values
df.dropna(inplace=True)
#removing unnecessary columns 
df.drop(['id', 'imdb_id','cast','homepage', 'tagline','keywords', 'overview','production_companies','release_date','director','genres','budget', 'revenue'], axis=1, inplace = True) 
#renaming column
df.rename(columns={'original_title':'title'}, inplace=True)
#confirming corrections
df.head(1)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis   Revealing complexity and patterns in the data   >Examine summary statistics >Examine visuals>Compare distributions >Describe correlations<jupyter_code># checking summary statistics for each column 
df.describe().astype(np.int64)<jupyter_output><empty_output><jupyter_text>>The summary statistics above gives a short overview of what is in the data set. It conveys a small but important description.    >It is communicating that>'popularity' column has 50% of data missing with MAX=32 and MIN=0;>'budget_adj' and 'revenue_adj' columns have 25% of data missing.<jupyter_code># exploring bar chart to check relationship between 'release_year' and 'revenue_adj'
df.groupby('release_year')['revenue_adj'].sum().plot(kind='bar',figsize= (25,5))
#exploring bar chart to check relationship between 'release_year' and 'popularity'
df.groupby('release_year')['popularity'].sum().plot(kind='bar',figsize= (25,5))<jupyter_output><empty_output><jupyter_text>>The bar charts above compare popularity average and revenue average from 1961 to 2015 year. It represents the >economic history (the rise) of the global film industry. In the given time slot, the most commercially successful >peaks in film industry with high revenue and high popularity rate were in 2011 and 2015.<jupyter_code># exploring scatter plot to check relationship between 'budget_adj' and 'revenue_adj'
df.plot(kind='scatter',x='budget_adj',y='revenue_adj',color='red', figsize= (8,8))
plt.show()<jupyter_output><empty_output><jupyter_text>>The scatter plot above is conveying strong positive relationship between 'revenue_adj' feature (the vertical axis) and 'budget_adj' feature (the horizontal axis). Meaning that most high-budget movies generate high revenue. However, a few extreme values suggest that in some rare cases the opposite is true when low-budget movies earn high revenue and high-budget movies experience great loss in revenue.<jupyter_code>#exploring relationship between 7 columns
plt.figure(figsize=(20,5))
sn.heatmap(df.corr(),annot=True,linewidths=10, cmap="coolwarm")
plt.xticks(rotation=45)
plt.title("Heatmap of Correlations")
#calculating correlation between features
df.corr()<jupyter_output><empty_output><jupyter_text>>The most strongest and meaningful relationship (0.77) can be noticed between 'vote_count' and 'popularity' features. >Also there are strong relationships between the following features:  0.74 between 'vote_count' and 'revenue_adj';  0.67 between 'budget_adj' and 'revenue_adj';  0.64 between 'budget_adj' and 'vote_count';  0.6 between 'popularity' and 'revenue_adj';    0.5 between 'popularity' and 'budget_adj'.<jupyter_code>#histograms below represents frequency distribution of data for seven choosen columns
df.hist(figsize=(10,10));<jupyter_output><empty_output><jupyter_text>>According seven histograms above, only 'vote_average' has a bell shape with one peak at value of 6 and with the minimum value of 2 and the maximum value of 8. > >'Vote_count', 'revenue_adj', 'budget_adj' distributions are right-skewed. >>'Release_year' distribution is left-skewed. >>'Runtime' distribution has gaps.<jupyter_code># plotting box plots for vote_average
df['vote_average'].plot(kind='box');<jupyter_output><empty_output><jupyter_text>>The boxplot above represents five-figure statistical summary of data in 'vote_average' column. The boxplot extends from the lower quartile, at the lower part of the box, to the greater quartile, the highest part. The length of the boxplot is interquartile range. >   >Also it shows the maximum value 8, average value 6 and the minimum value 2.<jupyter_code># removing all rows with missing values representing by zero
df2 = df[(df != 0).all(1)]
# calculating net profit
df2.loc[:,'net_profit'] = df2['revenue_adj'] - df2['budget_adj']
# converting floating numbers to integers
np.int64(df2['net_profit'])
# removing negative numbers
df3 = df2.loc[df2['net_profit']>0]
# calculating net profit percentage
df3.loc[:,'margin'] = (df3['net_profit'] / df3['revenue_adj'])*100
# calculating the number of remaining rows and columns
df3.shape<jupyter_output><empty_output><jupyter_text>Research Question 1 "What are TOP 100 most profitable movies ?"<jupyter_code># exploring the most profitable movies
#grouping results by two columns
df3.groupby('title').sum().sort_values('margin', ascending=False).head(3)
#creating 'profitability' data frame
#computing summary statistics
profitability = df3.groupby('title').sum().sort_values('margin', ascending=False).head(100)
profitability.describe()
#plotting bar chart of the most profitable movies
#grouping results by two columns
profitability.groupby('title')['margin'].sum().plot(kind='bar', figsize= (25,5))<jupyter_output><empty_output><jupyter_text>>The Number One Movie with the most profit is 'Halloween' with 181% of net margin ratio during the 55-year period.>>The second best movie - 'The Fog' with 156% of net margin ratio.>>The third best movie - 'HalloweenII' with 152% of net margin ratio.          Research Question 2 " What are TOP 100 most popular movies ?"<jupyter_code># exploring the most popular movies
df3.groupby('title').sum().sort_values('popularity', ascending=False).head(3)
#creating 'popularity' data frame
#computing summary statistics
popularity= df3.groupby('title').sum().sort_values('popularity', ascending=False).head(100)
popularity.describe()
# plotting bar chart of the most popular movies
#grouping results by two columns
popularity.groupby('title')['popularity'].sum().plot(kind='bar',figsize= (25,5))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09395973154362416, 'fraction_numerical': 0.020828511918537376, 'mean_word_length': 6.601583113456464, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 100, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '421710', 'license': 'MIT', 'n_tokens_mistral': 2593, 'n_tokens_neox': 2390, 'n_words': 953, 'path': 'Investigate_TMDb.ipynb', 'repo_name': 'reallygooday/investigate-database', 'size': '8657'}