<jupyter_start><jupyter_text>Project: Investigate a Dataset - [TMDB-movies] Table of ContentsIntroductionData WranglingExploratory Data AnalysisConclusions  Introduction > Dataset Description>> For my Data Analysis Project, I will be working on the 'tmdb-movies' dataset. I chose this dataset because I am a huge fan of hollywood movies. This dataset contains more than ten thousand movies and contains valuable information about those movies including budget, revenue, and profit. >  My questions for Analysis>> I will seek to answer the following questions with my Analysis today:   Which movie was the most and least profitable?    Which movie had the longest and the shortest runtime?    Which movie had the highest and the lowest budget?     Which movie was released the latest and the earliest?  >> I will also analyze the genre for successful movies and determine if directors have influence over profitability of the movies with visualizations.<jupyter_code># This cell is used to set up import statements for all of the packages that I plan to use for my analysis.

import pandas as pd
import numpy as np
from datetime import datetime
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>file:///Users/sovitkhanal/Library/Jupyter/runtime/jpserver-1381-open.html Data WranglingIn this section of the report, I will load the csv file dataset. I will also check for cleanliness of the data. Then I will trim and clean the dataset for analysis.  Data Load<jupyter_code>df = pd.read_csv('tmdb-movies.csv')    #Load the data and store it in 'df'<jupyter_output><empty_output><jupyter_text>Display the total number of rows and columns in the dataset<jupyter_code>df.shape          #Display the total number of rows and columns in the dataset<jupyter_output><empty_output><jupyter_text>Display the first five rows from the dataset<jupyter_code>df.head(5)                             #Display the contents of the first five rows<jupyter_output><empty_output><jupyter_text>Display the last five rows from the dataset<jupyter_code>df.tail(5)             #Display the contents of the last five rows<jupyter_output><empty_output><jupyter_text>Data CleaningIn this section I will perform data cleanup. I will display the data types, ensure the data type matches with what is shown. I will remove the duplicates, change the data types if required and also remove unnecessary data.   Display Data Types<jupyter_code>df.info()          #Display data types and check for to see if it needs to be cleaned<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 21 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   id                    10866 non-null  int64  
 1   imdb_id               10856 non-null  object 
 2   popularity            10866 non-null  float64
 3   budget                10866 non-null  int64  
 4   revenue               10866 non-null  int64  
 5   original_title        10866 non-null  object 
 6   cast                  10790 non-null  object 
 7   homepage              2936 non-null   object 
 8   director              10822 non-null  object 
 9   tagline               8042 non-null   object 
 10  keywords              9373 non-null   object 
 11  overview              10862 non-null  object 
 12  runtime               10866 non-null  int64  
 13  genres                10843 non-null  object 
 14  production_companies  9836 non-null   object 
 15  release_date       [...]<jupyter_text>Duplicate values (will be removed in the Data Cleanup Step)<jupyter_code>sum(df.duplicated())          #check if the data has duplicate records and print the number of duplicate records<jupyter_output><empty_output><jupyter_text>Investigate data type for release_date column (because it is listed as 'Object')<jupyter_code>print (type(df['release_date'][0])) #display the data type for release_date
if type(df['release_date'][0])!=object:
        print ("Hmmm...Interesting Find! The data type for release_date is not 'Object'.")<jupyter_output><class 'str'>
Hmmm...Interesting Find! The data type for release_date is not 'Object'.<jupyter_text>Other observations from the data setNo unit of currency is mentioned, we will assume it is US Dollar for our analysis.  Steps for Data CleanupRemove unnecessary columns. The columns we will remove are:    id, imdb_id, popularity, budget_adj, revenue_adj, homepage, production_company,, keywords, overview,         production_companies, vote_count, vote_average) Change the data type of release_date column to proper date formatRemove duplicate valuesReplace 0 with NaN in the runtime, budget and revenue columnsUpdate the format of budget and revenue columns  Remove unnecessary columns<jupyter_code>toDelete = [ 'id', 'imdb_id', 'budget_adj', 'revenue_adj', 'homepage', 
           'keywords', 'overview', 'production_companies', 'vote_count', 'vote_average']   #creates a list of columns

df = df.drop(toDelete, axis=1)       #Deletes the list with the columns
df.head(5)       #Display the first 3 rows of the new dataset<jupyter_output><empty_output><jupyter_text>Change the data type of release_date column to proper date format and then display the data<jupyter_code>df.release_date = pd.to_datetime(df['release_date'])      #Changes the data type of release_date column
df.head(5)     # Displays the new dataset with the data type updated<jupyter_output><empty_output><jupyter_text>Remove duplicate values<jupyter_code>df.drop_duplicates(keep ='first', inplace=True)
rows, col = df.shape

print('{} total entries remain in the dataset after removing the duplicates.'.format(rows, col))<jupyter_output>10865 total entries remain in the dataset after removing the duplicates.<jupyter_text>Replace 0 with NaN in the runtime, budget and revenue columns (to make it consistent; NaN = Not a Number)<jupyter_code>newList=['runtime', 'budget', 'revenue']     #Create list of runtime, budget, revenue columns

df[newList] = df[newList].replace(0, np.NAN)     #Replace '0' with NAN in the list

df.dropna(subset = newList, inplace = True)     #Delete all rows with NaN value<jupyter_output><empty_output><jupyter_text>Update the format of budget and revenue columns<jupyter_code>newType=['budget', 'revenue']     #Creates a list of budget and revenue  columns

df[newType]=df[newType].applymap(np.int64)     #Changes the data type of the new list

df.dtypes     #Displays the columns and the data type<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis  Because we want to calculate the profit, we need to calculate it based on the tota lbudget and revenue<jupyter_code>df.insert(2,'profit',df['revenue']-df['budget'])   #Insert a new column called 'Profit'. Calculate 'Profit' as a difference of 'Budget' and 'Revenue'
df.head(2)     #Display the new dataset<jupyter_output><empty_output><jupyter_text>For our analysis, we will create a function that will be used to calculate the high-low values<jupyter_code>import pprint

def highLow(column):       #Create and deifne the function

    high = df[column].idxmax()     #Highest value in the dataset
    detailedHigh = pd.DataFrame(df.loc[high])
    
    low = df[column].idxmin()      #Loswet value in the dataset
    detailedLow = pd.DataFrame(df.loc[low])
    
    details= pd.concat([detailedHigh, detailedLow], axis=1)     #Combining the records
    
    return details<jupyter_output><empty_output><jupyter_text>Research Question 1: Which movie made the highest and the lowest profit?<jupyter_code>highLow('profit')     #Calling the highLow functtion to retrieve highest and lowest profitable movies<jupyter_output><empty_output><jupyter_text>From the above analysis we can conclude the following about the most and least profitable movies:> The most profitable movie was 'Avatar' released in 2009 with a profit of \$254.5M.> The least profitable movie was 'The Warrior's Way' release in 2010 with a profit of -$413M.   Research Question 2: Which movie had the longest and the shortest runtime?<jupyter_code>highLow('runtime')     #Calling the highLow function to retrieve movies with longest and shortest runtime<jupyter_output><empty_output><jupyter_text>From the above analysis we can conclude the following about the movies with the longest and the shortest runtimes:> The longest movie was 'Carlos' released in 2010 with a runtime of 338 minutes.> The shortest movie was 'Kid's Story' released in 2003 with a runtime of 15 minutes.   Research Question 3: Which movie had the highest and the lowest budget?<jupyter_code>highLow('budget')     #Calling the highLow function to retrieve the highest and the lowest budgeted movies<jupyter_output><empty_output><jupyter_text>From the above analysis we can conclude the following about the movies with the highest and the lowest budget:> The movie with the highest budget was 'The Warrior's Way' which made a total profit of -\$413M.> The movie with the lowest budget was 'Lost & Found' which made a total profit of $99.  Research Question 4: Which movie was released the latest and the earliest?<jupyter_code>highLow('release_year')     #Calling the highLow function to retrieve the earliest and the latest release year<jupyter_output><empty_output><jupyter_text>From the above analysis we can conclude the following about the movies with the earliest and the latest release date> The latest movie released was 'Jurassic World'. The actual release date was June 9, 2015. > The earliest movie released was 'Psycho'. The actual release date was August 14, 1960.  Visualizations For the visualization, I will be plotting a histogram of the runtime of movies. This is helpful in figuring average runtime of the movies<jupyter_code>plt.figure(figsize=(10,4), dpi = 60)     #Indicates the width and height of the Histogram 

plt.xlabel('Runtime', fontsize = 20)     #Information for the X-axis

plt.ylabel('Number of Movies', fontsize=20)     #Information for the Y-axis

plt.title('Runtime with respect to Number of Movies', fontsize=20)     #Name of the Histogram

plt.hist(df['runtime'], rwidth = 0.8, bins = 40)     #Plot for the Histogram

plt.show()     #Display the Histogram<jupyter_output><empty_output><jupyter_text>Runtime Analysis<jupyter_code>df['runtime'].describe()    #Displays the runtime details<jupyter_output><empty_output><jupyter_text>Further exploration indicates that:  25% of the movies have runtime of about 95 minutes 75% of movies have runtime of about 119 minutes  Most Profitable Year<jupyter_code>profitYear = df.groupby('release_year')['profit'].sum()     #Find the most profitable year
print('The most profitable year was {}.'.format(profitYear.idxmax()))  #Display the most profitable Year<jupyter_output>The most profitable year was 2015.<jupyter_text>Movies with profit over $100M<jupyter_code>profits = df[df['profit'] >= 100000000]     #Only retrieve movies that had a profit of >$100M

profits.index = range(len(profits))
profits.index = profits.index + 1     #re-index

profits.head(5)     #Display the new list (top 5)
print('There are a total of {} movies with profit over $100M.'.format(len(profits))) # Display the total number of movies with profit >$100M<jupyter_output>There are a total of 826 movies with profit over $100M.<jupyter_text>Function to take any column as argument and display results<jupyter_code>def anyColumn(column):     #Create a new function that takes any column as argument and display results
    anyColumn = profits[column].str.cat(sep = '|')     #separate the string from the column with '|'
    
    anyColumn = pd.Series(anyColumn.split('|'))     #store values separately in a series
    
    total = anyColumn.value_counts(ascending = False)     #arrange values in descending order
    
    return total<jupyter_output><empty_output><jupyter_text>Display number of profitable movies by Genre<jupyter_code>myVals = anyColumn('genres')     #store returned value in a variable

myVals.head()     #Display the count of top 5 values<jupyter_output><empty_output><jupyter_text>Most frequent Genres in a visual format The most frequently used genre is Action followed by Comedy. This can be depicted in a pie chart below.<jupyter_code>myVals.sort_values(ascending = True, inplace = True)     #Plotting the points in descending order

pt = myVals.plot(kind="pie",autopct="%1.1f%%")  #Plotting

pt.set(title = 'Most frequent Genres with respect to Number of Movies')     #Give our chart a title

pt.figure.set_size_inches(10, 5)     #Size of the chart<jupyter_output><empty_output><jupyter_text>Number of profitable movies by Director<jupyter_code>myVals = anyColumn('director')
myVals.head()<jupyter_output><empty_output><jupyter_text><NAME> was one of the most successful directors. From the data that we gathered, we can conclude that movies directed by Steven make profits greater than $100M and the director does have influence over profitability.   Does a higher budget movie end being popular than a low budget movie?<jupyter_code># Scatter plot for two variables
f, ax = plt.subplots(figsize=(10,5))
ax.set(ylim=(0, 10))
sns.regplot(x=df["budget"],y=df["popularity"]).set_title("Budget Vs Popularity",size=15)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07607955834530741, 'fraction_numerical': 0.020040837933903047, 'mean_word_length': 4.42189421894219, 'pattern_counts': {'":': 0, '<': 111, '<?xml version=': 0, '>': 130, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '460905', 'license': 'Apache-2.0', 'n_tokens_mistral': 3970, 'n_tokens_neox': 3685, 'n_words': 1493, 'path': 'Investigate a Dataset.ipynb', 'repo_name': 'typewriterrr/Udacity-Data-Analysis-Investigate-a-Dataset', 'size': '13233'}