<jupyter_start><jupyter_text>Project: Investigate TMDb movie data Table of ContentsIntroductionData WranglingExploratory Data AnalysisConclusions  IntroductionThe data set I choose is the TMDb movie data set, which contains more than 10,000 movies collected from The Movie Database. This project will focus on the numeric data and their relationships, including revenue, budget, popularity, and vote rating of these movies. Import the packages:<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from datetime import datetime
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Data Wrangling General Properties Let's have a glimpse of our data:<jupyter_code>df = pd.read_csv("tmdb-movies.csv")
df.head()<jupyter_output><empty_output><jupyter_text>We can see that cast, genres, and production companies columns contains multiple values separated by '|'. Besides, the release date column has a pattern of month/day/year.<jupyter_code>df.genres.head()  
df.genres.tail(10)
df.production_companies.head()<jupyter_output><empty_output><jupyter_text>We can just leave them here, explore the shape and other attributes of the data set first. We will deal with these columns later. Check the shape of our data:<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>Check basic information and nulls:<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 21 columns):
id                      10866 non-null int64
imdb_id                 10856 non-null object
popularity              10866 non-null float64
budget                  10866 non-null int64
revenue                 10866 non-null int64
original_title          10866 non-null object
cast                    10790 non-null object
homepage                2936 non-null object
director                10822 non-null object
tagline                 8042 non-null object
keywords                9373 non-null object
overview                10862 non-null object
runtime                 10866 non-null int64
genres                  10843 non-null object
production_companies    9836 non-null object
release_date            10866 non-null object
vote_count              10866 non-null int64
vote_average            10866 non-null float64
release_year            10866 non-null int64
budget_adj              1[...]<jupyter_text>Check the unique numbers of the data set:<jupyter_code>df.nunique()
df.describe()<jupyter_output><empty_output><jupyter_text>We can see that nearly half of the movies has no budget and no revenue. We may need to dig into these observations to see what is really going on.<jupyter_code>df.query('budget == 0').head()
df.query('budget == 0').info()
df.query('revenue == 0').info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 6016 entries, 48 to 10865
Data columns (total 16 columns):
id                      6016 non-null int64
popularity              6016 non-null float64
budget                  6016 non-null int64
revenue                 6016 non-null int64
original_title          6016 non-null object
director                5973 non-null object
keywords                6016 non-null object
runtime                 6016 non-null int64
genres                  6016 non-null object
production_companies    6016 non-null object
release_date            6016 non-null object
vote_count              6016 non-null int64
vote_average            6016 non-null float64
release_year            6016 non-null int64
budget_adj              6016 non-null float64
revenue_adj             6016 non-null float64
dtypes: float64(4), int64(6), object(6)
memory usage: 799.0+ KB<jupyter_text>These movies looks normal, I can't really get a hint from them, the zero budget may be a kind of missing value, or it may has no budget at the first place. Anyway, since I have no information how these zeros are from, I just gonna keep the data as it was.  Data Cleaning (Replace this with more specific notes!) For now, we can see that there is a column named keyword that has not been checked in the head function, we need to have a look at it.<jupyter_code>df.keywords.head()<jupyter_output><empty_output><jupyter_text>We can see that in genres, keywords, and production companies columns, the strings in each column is not alphabetic ordered, they are more of a importance order. Besides, we can see that some of these columns have only one string. So, I will cut off the others, leave only the first string in these three columns.<jupyter_code>trim_columns = ['keywords', 'genres', 'production_companies']

for col in trim_columns:
    df[col] = df[col].astype(str)

for col in trim_columns:
    df[col] = df[col].apply(lambda x: x.split('|')[0])
df.head()[['keywords','genres','production_companies']]<jupyter_output><empty_output><jupyter_text>We can check how many unique values in these columns:<jupyter_code>df.nunique()[['keywords','genres','production_companies']]<jupyter_output><empty_output><jupyter_text>Drop some columns that we are not going to analyze:<jupyter_code>df.drop(["imdb_id", "cast", "homepage", "tagline", "overview"], axis=1, inplace=True)
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 16 columns):
id                      10866 non-null int64
popularity              10866 non-null float64
budget                  10866 non-null int64
revenue                 10866 non-null int64
original_title          10866 non-null object
director                10822 non-null object
keywords                10866 non-null object
runtime                 10866 non-null int64
genres                  10866 non-null object
production_companies    10866 non-null object
release_date            10866 non-null object
vote_count              10866 non-null int64
vote_average            10866 non-null float64
release_year            10866 non-null int64
budget_adj              10866 non-null float64
revenue_adj             10866 non-null float64
dtypes: float64(4), int64(6), object(6)
memory usage: 1.3+ MB<jupyter_text>Check missing values:<jupyter_code>df.isnull().sum(axis=0)<jupyter_output><empty_output><jupyter_text>Only director column has 44 missing values.  Exploratory Data Analysis Discover the distributions of some columns which contains numeric values.<jupyter_code>df.hist(figsize=(15, 10));<jupyter_output><empty_output><jupyter_text>We can see that budget, popularity, revenue, runtime, and vote_count are right skewed. Vote_average is nearly a normal distribution.We can also check the scatter plots of budget, vote, revenue, and popularity to see if they have any connections.<jupyter_code>plt.scatter(df.budget_adj, df.revenue_adj)
plt.xlabel("budget")
plt.ylabel("revenue");<jupyter_output><empty_output><jupyter_text>Seems like there is a weak and positive correlation between budget and revenue.<jupyter_code>plt.scatter(df.budget_adj, df.popularity)
plt.xlabel("budget")
plt.ylabel("popularity");<jupyter_output><empty_output><jupyter_text>Seems like there is a weak and positive correlation between budget and popularity.<jupyter_code>plt.scatter(df.popularity, df.revenue_adj)
plt.xlabel("popularity")
plt.ylabel("revenue");<jupyter_output><empty_output><jupyter_text>This plot shows a medium positive correlation between popularity and revenue.<jupyter_code>plt.scatter(df.vote_average, df.revenue)
plt.xlabel("vote")
plt.ylabel("revenue");<jupyter_output><empty_output><jupyter_text>Very weak correlation between vote and revenue.  Does the average vote of movies differ in different decades? Seeing all these data, the first question I think about is that is there a trend in the vote of movies in different decades?<jupyter_code>vote_avg_decade = df.groupby((df.release_year//10)*10).mean().vote_average
vote_avg_decade
locations = [1, 2, 3, 4, 5, 6]
heights = vote_avg_decade
labels = ["1960s", "1970s", "1980s", "1990s", "2000s", "2010s"]
plt.bar(locations, heights, tick_label = labels)
plt.title("Average vote rating scores of decades")
plt.xlabel("Decades")
plt.ylabel("Average vote rating scores");<jupyter_output><empty_output><jupyter_text>From the bar plot, we can see that   Which genre of movie has the highest revenue on average? As I keep thinking, which of the genres can achieving the highest revenue?<jupyter_code>avg_revenue_genres = df.groupby(['genres'])['revenue_adj'].mean()
avg_revenue_genres
plt.figure(figsize=(15,10))
avg_revenue_genres.sort_values(ascending=False).plot.bar()
plt.title("Average revenue for different genres")
plt.xticks(rotation=50)
plt.xlabel("Genres of movie")
plt.ylabel("Average revenue");<jupyter_output><empty_output><jupyter_text>We can also use box plot to dig into this problem a little deeper:<jupyter_code>df_rev = df[['revenue_adj','genres']]
df_rev.boxplot(by = 'genres', figsize=(15,10),rot=50);<jupyter_output><empty_output><jupyter_text>We can see that many genres have a median of zero, that may because half of the movies has a budget or revenue of 0.  Which genre of movie has the highest popularity on average? We can also explore the relationship between popularity and genres of movie:<jupyter_code>avg_popularity_genres = df.groupby(['genres'])['popularity'].mean()
avg_popularity_genres
plt.figure(figsize=(15,10))
avg_popularity_genres.sort_values(ascending=False).plot.bar()
plt.title("Average popularity for different genres")
plt.xticks(rotation=50)
plt.xlabel("Genres of movie")
plt.ylabel("Average popularity");<jupyter_output><empty_output><jupyter_text>The top 3 popular genres of movie are adventure, science fiction, and fantasy, which is the same as the top 3 places in the highest revenue plot. This finding just matches the correlation between popularity and revenue we found in the scatter plot.  Which genre of movie has the highest vote?<jupyter_code>avg_vote_genres = df.groupby(['genres'])['vote_average'].mean()
avg_vote_genres
plt.figure(figsize=(15,10))
avg_vote_genres.sort_values(ascending=False).plot.bar()
plt.title("Average vote for different genres")
plt.xticks(rotation=50)
plt.xlabel("Genres of movie")
plt.ylabel("Average vote");<jupyter_output><empty_output><jupyter_text>This plot is different from both revenue and popularity plot. People tend to give high vote to documentary, music, and animation genres, instead of adventrue, science fiction, and fantasy genres.  Which production company has the highest revenue on average? After thinking about genres of movies, we can also check which production company in the data set is the one that can make the highest revenue on average.<jupyter_code>avg_revenue_genres = df.groupby(['genres'])['revenue_adj'].mean()
plt.figure(figsize=(15,10))
avg_vote_genres.sort_values(ascending=False).plot.bar()
plt.title("Average vote for different genres")
plt.xticks(rotation=50)
plt.xlabel("Genres of movie")
plt.ylabel("Average vote");
avg_revenue_companies = df.groupby(['production_companies'])['revenue_adj'].mean()
avg_revenue_companies.sort_values(ascending=False).head(20)
plt.figure(figsize=(15,10))
avg_revenue_companies.sort_values(ascending=False).head(20).plot.bar()
plt.title("Average revenue of production companies")
plt.xticks(rotation=50)
plt.xlabel("Production companies")
plt.ylabel("Average revenue");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08750110258445797, 'fraction_numerical': 0.03960483373026374, 'mean_word_length': 4.5092322643343055, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 93, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '48570', 'license': 'MIT', 'n_tokens_mistral': 3729, 'n_tokens_neox': 3316, 'n_words': 1097, 'path': 'investigate_a_dataset.ipynb', 'repo_name': 'ZhenghaoXiao32/data_investigation_udacity', 'size': '11337'}