<jupyter_start><jupyter_text>Project 2: Investigate a TMDb Movie Dataset Table of ContentsIntroductionData WranglingExploratory Data AnalysisConclusions  IntroductionThis project is a part of Udacity Data Analyst Nanodegree. The goal of this project is to go through the data analysis process and see how everything fits together. I have chosen to investigate the TMDb movie dataset which is provided by Udacity and originate from Kaggle. This dataset contains information about 10,000 movies collected from The Movie Database (TMDb), including user ratings and revenue. Questions1. Which genres are most popular from year to year?2. What kinds of properties are associated with movies that have high revenues?3. Did movies with higher budget received a better rating?4. Which year have most of high popularity movies?  Import all packages needed for this analysis1. os for managing the file path.2. NumPy for array and matrix operation.3. Pandas for data analysis.4. Matplotlib and seaborn for vasualization.<jupyter_code># Import statements for all of the packages that are considered to be used
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
sns.set_style("whitegrid")

# Display float number instead of scientific notation
pd.options.display.float_format = '{:.2f}'.format<jupyter_output><empty_output><jupyter_text>Data WranglingThis process is to assess the data to identify any problems in the data’s quality or structure, and clean the data by modifying, replacing, or removing data to ensure that the dataset has the highest quality and as well-structured as possible. General Properties The first step is to read the data from csv file and stored it in a pandas dataframe called df.<jupyter_code># Load the data from csv file and stored it to a variable
df = pd.read_csv(os.path.join('data', 'tmdb-movies.csv'))<jupyter_output><empty_output><jupyter_text>Preview first 5 rows of the dataset in order to understand what the data looks like.<jupyter_code># Preview first 5 rows of the dataset.
df.head()<jupyter_output><empty_output><jupyter_text>Regarding the above table, I noticed that `cast`, `director`, `genres` and `production_companies` column has multiple values separated by pipe in the same row so I have to separate these values into many columns to make it easier to analyze the data. Then find the basic information of the dataset to check null values and data types of each column.<jupyter_code># Let's see the column info, null values and data types
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 21 columns):
id                      10866 non-null int64
imdb_id                 10856 non-null object
popularity              10866 non-null float64
budget                  10866 non-null int64
revenue                 10866 non-null int64
original_title          10866 non-null object
cast                    10790 non-null object
homepage                2936 non-null object
director                10822 non-null object
tagline                 8042 non-null object
keywords                9373 non-null object
overview                10862 non-null object
runtime                 10866 non-null int64
genres                  10843 non-null object
production_companies    9836 non-null object
release_date            10866 non-null object
vote_count              10866 non-null int64
vote_average            10866 non-null float64
release_year            10866 non-null int64
budget_adj              1[...]<jupyter_text>Regarding the above information, I got the interesting information of the dataset as following. - there are total 10,866 rows and total 21 columns. - The data type of each column is reasonable except `release_date` that should be **datetime** instead of **int64** but I decided to drop it because I will not use it to answer research questions. - There are **many null values** in `homepage`, `tagline`, `keywords` and `production_companies` column but these columns are not necessary to answer the research questions so I decided to drop these columns.- there are **few null values** in `cast`, `director` and `genres` column, and because these columns need to be used to answer the research questions so I decide to drop only those rows with null values. - Moreover, I decided to drop `id`, `imdb_id`, `overview`, `vote_count` and `vote_average` columns since these columns are not used to answer the reserach questions. Then let's see the basic statistics and histograms of the dataset.<jupyter_code># Let's see the basic statistics of the data
df.describe()
# See all histogram of each column
df.hist(figsize=(12,12))
plt.show()<jupyter_output><empty_output><jupyter_text>Regarding the above table and histograms, I noticed that there are **many zero values** in `budget`, `revenue`, `runtime`, `budget_adj` and `revenue_adj` columns that doesn't make sense because every movies should have budget, revenue and runtime. I assume that values in these columns maybe missing so I decided to replace "0" values with "NaN". For budget and revenue on this research I decided to use `budget_adj` and `revenue_adj` column because they have already adjusted inflation rate over the time period so I will drop both `budget` and `revenue` column since I will not use it.  Data CleaningAfter exploring the data in the data wragling process, I concluded the data cleaning as following.- Drop unused columns (id, imdb_id, budget, revenue, homepage, tagline, keywords, overview, production_companies, release_date, vote_count and vote_average).- Because budget_adj and revenue_adj have already adjusted inflation rate so I chose to use these 2 columns instead of budget and revenue columns.- Change 0 values in budget, revenue and runtime column to NaN.- Drop duplicated values.- Drop null values.- Manage columns with muliple values separated by pipe. (cast, director and genres) Drop unused columns.<jupyter_code># Drop unused columns.
df.drop(['id',
         'imdb_id',
         'budget',
         'revenue',
         'homepage',
         'tagline',
         'keywords',
         'overview',
         'production_companies',
         'release_date',
         'vote_count',
         'vote_average'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Let's see the data after drop unsed columns.<jupyter_code># Let see first five rows of the data after drop unused columns.
df.head()<jupyter_output><empty_output><jupyter_text>The unused columns have been dropped completely. Then replace 0 values with NaN.<jupyter_code># Replace 0 value with NaN
df['revenue_adj'].replace(0, np.NAN, inplace=True)
df['budget_adj'].replace(0, np.NAN, inplace=True)
df['runtime'].replace(0, np.NAN, inplace=True)
# Check whether revenue_adj column has 0 value or not
df.query('revenue_adj == 0')<jupyter_output><empty_output><jupyter_text>`revenue_adj` column no longer has 0 value.<jupyter_code># Check whether budget_adj column has 0 value or not
df.query('budget_adj == 0')<jupyter_output><empty_output><jupyter_text>`budget_adj` column no longer has 0 value.<jupyter_code># Check whether runtime column has 0 value or not
df.query('runtime == 0')<jupyter_output><empty_output><jupyter_text>`runtime` column no longer has 0 value. Check all null values in the dataset.<jupyter_code># See null values
df.isna().sum()<jupyter_output><empty_output><jupyter_text>Regarding the above information, there are **few null values** in `cast`, `director`, `runtime` and `genres`. And there are **quite a lot of null values** in `budget_adj` and `revenue_adj` so let's drop them in the next cell. Drop null values in the dataset.<jupyter_code># Drop null values
df.dropna(inplace=True)
# Confirm that the dataset has no null value
df.isna().sum()<jupyter_output><empty_output><jupyter_text>Now the dataset no longer has a null value. Then let's see duplicated values in the dataset.<jupyter_code># See duplicated values
df.duplicated().sum()<jupyter_output><empty_output><jupyter_text>There is 1 duplicated value in the dataset so let's drop it in the next cell.<jupyter_code># Drop duplicated values
df.drop_duplicates(inplace=True)
# Confirm that the dataset has no duplicated value
df.duplicated().sum()<jupyter_output><empty_output><jupyter_text>Now the dataset no longer has a duplicated value. Let's see the information of the dataset after cleaning.<jupyter_code># See information of the dataset after cleaning
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 3849 entries, 0 to 10848
Data columns (total 9 columns):
popularity        3849 non-null float64
original_title    3849 non-null object
cast              3849 non-null object
director          3849 non-null object
runtime           3849 non-null float64
genres            3849 non-null object
release_year      3849 non-null int64
budget_adj        3849 non-null float64
revenue_adj       3849 non-null float64
dtypes: float64(4), int64(1), object(4)
memory usage: 300.7+ KB<jupyter_text>The above information shown the cleanness of the dataset. Finally, let's deal with the columns that contain multiple data at the same row in the next cell. Manage columns with muliple values separated by pipe by use split method to separate values in to many columns.<jupyter_code># Manage columns with muliple values separated by pipe by use split method to separate values in to many columns.
df_cast = df['cast'].str.split('|', expand=True).rename(columns=lambda x: "cast_{}".format(x + 1))
df_director = df['director'].str.split('|', expand=True).rename(columns=lambda x: "director_{}".format(x + 1))
df_genres = df['genres'].str.split('|', expand=True).rename(columns=lambda x: "genres_{}".format(x + 1))<jupyter_output><empty_output><jupyter_text>Then join the separated columns with the original dataframe and preview first five rows of the dataset.<jupyter_code># Join the separated columns with the original dataframe and see the preview data.
df = df.join([df_cast, df_director, df_genres])
df.drop(['cast', 'director', 'genres'], axis=1, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Regarding the above table, there are new columns that separate values in `cast`, `director` and `genres` columns into many columns as needed. Now I am ready to analyze the data.  Exploratory Data AnalysisAfter cleaning the data, Now I am ready to answer the research questions. Research Question 1 Which genres are most popular from year to year?  First step is to select all genres columns to join with release_year column and stored it in a new dataframe.<jupyter_code># Select all genres columns and release year column then join it together and then store it in a new dataframe
df_genres_years = df.loc[:, 'genres_1':'genres_5'].join(df['release_year'])<jupyter_output><empty_output><jupyter_text>Preview first 5 rows of the dataset in order to understand what the data looks like.<jupyter_code># See the first five rows of the dataset
df_genres_years.head()<jupyter_output><empty_output><jupyter_text>Then selects unique year from `release_year` column and store it in a variable call `years` and also creates an empty list call `most_popular` to store results of the most popular genre each year. After that, iterates into years to find the most popular genres each year using pandas value counts method and append them in the `most_popular` list. Finally, convert `most_popular` list to a new dataframe call `most_popular_genre`.<jupyter_code># Select unique year from release year column and store it in years variable
years = df_genres_years['release_year'].unique()

# creates a list of tuples for most popular genres in each year
most_popular = []

# Iterate into years
for year in years:
    # Creates new dataframe filterd by each year in years
    genres_by_year = df_genres_years[df_genres_years['release_year'] == year]
    
    # Drops release_years columns to performs value counts in the next line
    drop_year = genres_by_year.drop(['release_year'], axis = 1)
    
    # Performs value counts of genres and select index of maximum value and store it in 
    popular_genre_name = drop_year.stack().value_counts().idxmax()
    
    # Appends year and most popular genre each year in most_popular list
    most_popular.append((year, popular_genre_name))

# Creates new dataframe to shows most popular genre each year
most_popular_genre = pd.DataFrame(most_popular, columns=['release_year', 'popular_genre'])<jupyter_output><empty_output><jupyter_text>Let's see the most popular genre each year.<jupyter_code># Showing most popular genre each year
most_popular_genre.sort_values('release_year', ascending=False)<jupyter_output><empty_output><jupyter_text>Let's see how many times these genres are appeared in the most popular ranking.<jupyter_code># Most popular genres of all years
most_popular_genre['popular_genre'].value_counts()<jupyter_output><empty_output><jupyter_text>Answer Question 1 Regarding the above two tables, the most popular genre that has been produced each year is Drama which appeared in the ranking 37 times followed by Comedy 11 times and Thriller 3 times.  Research Question 2  What kinds of properties are associated with movies that have high revenues? First step is to see basic statistics of revenue_adj column to defind point of high revenues movie.<jupyter_code># See basic statistics of revenue_adj column
df['revenue_adj'].describe()<jupyter_output><empty_output><jupyter_text>As the above statistics, I define high revenue movies that have to earn more than 50% average of all movies or more than 61,813,934.33 US dollars. Let's see the histogram of revenue_adj column.<jupyter_code># See the histogram of revenue_adj column
fig, ax = plt.subplots(figsize=(8, 5))
sns.distplot(df['revenue_adj'])
ax.set_title("revenue_adj histogram", fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Regarding the above histogram, I noticed that the data are right-skewed, then the mean is typically greater than the median. Most movies have revenue less than 500 million. Select high revenue movies as I have mentioned above and saved it in a new dataframe.<jupyter_code># Define high revenue movies that have to earn more than the 50% average of all movies and store it in a new dataframe
df_high_revenue = df.query('revenue_adj > 61813934.33')<jupyter_output><empty_output><jupyter_text>Preview first 5 rows of the dataset in order to understand what the data looks like.<jupyter_code># See first five rows of the dataset
df_high_revenue.head()<jupyter_output><empty_output><jupyter_text>Let's calculated characteristics of high revenue movies as following.- Average polularity score of high revenue movies.- Average runtime of high revenue movies.- Average budget that high revenue movies spended.- Frequent director that appeared in high revenue movies.- Frequent cast that appeared in high revenue movies.- Popular genre that has been produced.- Release year of high revenue movies.<jupyter_code># Calculated characteristics of high revenue movies.

# Average Polularity
avg_popularity = df_high_revenue['popularity'].mean()

# Average Runtime
avg_runtime = df_high_revenue['runtime'].mean()

# Average Budget
avg_budget = df_high_revenue['budget_adj'].mean()

# Frequent director
director = df_high_revenue.loc[:, 'director_1': 'director_12']
frequent_director = director.stack().value_counts().idxmax()

# Frequent cast
cast = df_high_revenue.loc[:, 'cast_1': 'cast_5']
frequent_cast = cast.stack().value_counts().idxmax()

# Popular genre
genre = df_high_revenue.loc[:, 'genres_1': 'genres_5']
frequent_genre = genre.stack().value_counts().idxmax()

# Release year
release_year = df_high_revenue['release_year'].value_counts().idxmax()<jupyter_output><empty_output><jupyter_text>After done computing the characteristics of high revenue movies then I will store all characteristics in a new dataframe called `high_revenue_summary`.<jupyter_code># Create new dataframe to store characteristics of high revenue movies
high_revenue_summary = pd.DataFrame(data=[avg_popularity, avg_runtime, avg_budget, frequent_director, frequent_cast, frequent_genre, release_year], 
                                    index=['avg_popularity', 'avg_runtime', 'avg_budget', 'frequent_director', 'frequent_cast', 'frequent_genre', 'release_year'],
                                    columns=['description']
                                   )<jupyter_output><empty_output><jupyter_text>Let's see the characteristics of high revenue movies.<jupyter_code># See the characteristics of high revenue movies
high_revenue_summary<jupyter_output><empty_output><jupyter_text>Answer Question 2Regarding the above table, most high revenue movies tend to have average popularity at 1.70 points, average runtime at 113.63 minutes, average budget at 65,804,609.15 US dollars, director should be <NAME>, cast should included <NAME> and genre should be drama.  Research Question 3 Did movies with higher budget received a better rating? Firstly, let's see the histogram of budget_adj column.<jupyter_code># See the histogram of revenue_adj column
fig, ax = plt.subplots(figsize=(8, 5))
sns.distplot(df['budget_adj'])
ax.set_title("budget_adj histogram", fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Regarding to the above histogram, I noticed that the data are right-skewed, then the mean is typically greater than the median. Most movies spend the budget less than 100 million. Then let's see the basic statistics of popularity column.<jupyter_code># See basic statistics of popularity column
df['popularity'].describe()<jupyter_output><empty_output><jupyter_text>As the above statistics, I noticed that the max value of the data is significantly higher than other values so I define it as an outlier and I decided to ignore it later. Then let's see the histogram of popularity column.<jupyter_code>fig, ax = plt.subplots(figsize=(8, 5))
sns.distplot(df['popularity'])
ax.set_title("popularity histogram", fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Regarding to the above histogram, I noticed that the data are right-skewed, then the mean is typically greater than the median. Most movies have popularity less than 5 points. Then let's eliminates the outlier as I have mentioned above and saved the result to a new data frame called df_popularity.<jupyter_code># Eliminates outlier values of popularity column
df_popularity = df.query('popularity < 15')<jupyter_output><empty_output><jupyter_text>Preview first 5 rows of the dataset in order to understand what the data looks like.<jupyter_code>df_popularity.head()<jupyter_output><empty_output><jupyter_text>Then visualized relationship between budget_adj and popularity to see if movies with higher budget received a better rating.<jupyter_code># Visualized relationship between budget_adj and popularity
fig, ax = plt.subplots(figsize=(10, 8))
sns.regplot(ax=ax, x="budget_adj", y="popularity", data=df_popularity, line_kws={"color": "r"})
ax.set_title("Relationship between budget and popularity", fontsize=14)
ax.set_xlim(0, None)
plt.show()<jupyter_output><empty_output><jupyter_text>Answer Question 3Regarding the above graph, I seen that `budget_adj` has a positive relationship with `popularity`. Therefore, movies with high budgets also result in high popularity.  Research Question 4 Which year have most of high popularity movies? First step is to see basic statistics of popularity column to defind high popularity point.<jupyter_code># See basic statistics of popularity column
df['popularity'].describe()<jupyter_output><empty_output><jupyter_text>As the above statistics, I define high popularity point base on movies that have popularity more than 75% average of all movies or more than 1.37 points. Then selects all high popularity movies as I have mentioned above and saved it in a new dataframe called df_high_pop.<jupyter_code># Select high popularity movies (popularity more than 75% average of all movies)
df_high_pop = df.query('popularity > 1.37')<jupyter_output><empty_output><jupyter_text>Preview first 5 rows of the dataset in order to understand what the data looks like.<jupyter_code># See first five rows of the dataset
df_high_pop.head()<jupyter_output><empty_output><jupyter_text>Then counts high popularity movies and grouped by release year and then saved it in a new dataframe call pop_year.<jupyter_code># Counts high popularity each year
pop_year = df_high_pop.groupby('release_year')['popularity'].count().reset_index()<jupyter_output><empty_output><jupyter_text>Let's see preview of year with the most of high popularity movies.<jupyter_code>pop_year.sort_values('popularity', ascending=False).head()<jupyter_output><empty_output><jupyter_text>Becuase the number of movies each year is varied, I decided to use the proportion of high popularity movies each year instead. so that I have to count all movies each year.<jupyter_code># Counts movies each year
movies_count = df.groupby('release_year')['popularity'].count().reset_index()
movies_count.head()<jupyter_output><empty_output><jupyter_text>Then merges all movies counted and high popularity movies counted into a new dataframe called df_pop_year and calculated the propotion of high popularity movies and saved it in the new column called pop_propotion.<jupyter_code># Merges movies count each year and high popularity each year
df_pop_year = movies_count.merge(pop_year, on='release_year', suffixes=('', '_high'))

# Calculates the proportion of high popularity movies
df_pop_year['pop_propotion'] = df_pop_year['popularity_high'] / df_pop_year['popularity']<jupyter_output><empty_output><jupyter_text>Finally, let's see the table that shows which release year with the most high popularity movies.<jupyter_code># Shows ranking of high popularity movies sorted by the propotion of high popularity movies
df_pop_year.sort_values('pop_propotion', ascending=False).head()<jupyter_output><empty_output><jupyter_text>To clearly understand, let's visualized the propotion of high popularity movies by year.<jupyter_code># Visualized the propotion of high popularity movies by year
fig, ax = plt.subplots(figsize=(14, 6))
sns.lineplot(ax=ax, x="release_year", y="pop_propotion", label="pop_propotion", data=df_pop_year)
ax.set_title("The propotion of high popularity movies by year", fontsize=14)
plt.xticks(np.arange(min(df_pop_year['release_year']), max(df_pop_year['release_year'])+1, 5))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0765233611653935, 'fraction_numerical': 0.015810978859477704, 'mean_word_length': 5.279141104294479, 'pattern_counts': {'":': 1, '<': 180, '<?xml version=': 0, '>': 181, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '44', 'dir': 'jupyter-structured-clean-dedup', 'id': '240523', 'license': 'MIT', 'n_tokens_mistral': 6668, 'n_tokens_neox': 6213, 'n_words': 2570, 'path': 'investigate_a_dataset/investigate-a-dataset.ipynb', 'repo_name': 'phattp/data-analyst-udacity', 'size': '22532'}