<jupyter_start><jupyter_text>Project: TMDB-Movies Dataset Analysis Table of ContentsIntroductionData WranglingExploratory Data AnalysisConclusions  Introduction[TMDB-Movies](https://www.themoviedb.org/) is The movie database(TMDb). This database is community built movies and TV database. In this database have many movies and features, but I use following properties:    * popularity    * budget_adj    * revenue_adj    * keywords    * genres    * release_date    In this report, I explore the following questions:    * How has the profitability of making films changed over time?    * When do movies of each genre or keyword make the best profits?    Throughout my analysis film profitability and popularity will be dependent variables, while release date, budget, genres, and keywords will be independent variables.<jupyter_code># Use this cell to set up import statements for all of the packages that you
#   plan to use.
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

# Remember to include a 'magic word' so that your visualizations are plotted
#   inline with the notebook. See this page for more:
#   http://ipython.readthedocs.io/en/stable/interactive/magics.html
%matplotlib inline
#plt.style.use('seaborn')<jupyter_output><empty_output><jupyter_text>Data WranglingIn data wrangling step, I want to collect some meaningful columns and change to useful data types.* Remove unuseful columns* Change `release_date` format to `datetime* Split `genres` and `keywords` string by "|"* Fill mean value in NaN or 0 data General PropertiesFirst of all I want to what kind of data in this dataset. See some rows and described table.<jupyter_code>df = pd.read_csv("tmdb-movies.csv")
df.head()
df.describe()
df.info()
# Remove unusefule columns
df2 = pd.DataFrame(df, columns = ['popularity', 'budget_adj', 'revenue_adj', 'keywords', 'genres', 'release_date', 'release_year', 'original_title'])
df2.keys()
df3 = pd.DataFrame(df2)
print(df3['release_date'])
df3['release_date'] = pd.to_datetime(df2['release_date'])
print(df3['release_date'])
df3['release_date'] = df3.apply(lambda x: x['release_date'].replace(year = x['release_year']), axis = 1)
print(df3['release_date'])
# Split string data
df4 = pd.DataFrame(df3)

df4['keywords'] = df4['keywords'].str.split("|")
df4['genres'] = df4['genres'].str.split("|")

df2.head(3)<jupyter_output><empty_output><jupyter_text>Data CleaningDataset have some null data, therefore I clean some columns.<jupyter_code>df4.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 8 columns):
 #   Column          Non-Null Count  Dtype         
---  ------          --------------  -----         
 0   popularity      10866 non-null  float64       
 1   budget_adj      10866 non-null  float64       
 2   revenue_adj     10866 non-null  float64       
 3   keywords        9373 non-null   object        
 4   genres          10843 non-null  object        
 5   release_date    10866 non-null  datetime64[ns]
 6   release_year    10866 non-null  int64         
 7   original_title  10866 non-null  object        
dtypes: datetime64[ns](1), float64(3), int64(1), object(3)
memory usage: 679.2+ KB<jupyter_text>There are null data in `keywords` and `genres`. These columns cannot fill any estimated value(because these are string type), then I remove some rows which are included null data.<jupyter_code># remove null data
df5 = pd.DataFrame(df4)
df5.dropna(inplace = True)
df5.info()
df6 = pd.DataFrame(df5)
print(df6['budget_adj'].mean())
print(df6['revenue_adj'].mean())

df6.loc[df6['budget_adj'] == 0, 'budget_adj'] = df6['budget_adj'].mean()
df6.loc[df6['revenue_adj'] == 0, 'revenue_adj'] = df6['revenue_adj'].mean()

fig, ax = plt.subplots(2,1,figsize = (10, 8))
sns.histplot(data = df6['budget_adj'], bins = 50, fill = True, ax = ax[0], alpha = 0.5, label = 'after', color = 'red')
sns.histplot(data = df5['budget_adj'], bins = 50, fill = True, ax = ax[0], alpha = 0.5, label = 'before', color = 'blue')
sns.histplot(data = df6['revenue_adj'], bins = 50, fill = True, ax = ax[1], alpha = 0.5, color = 'red')
sns.histplot(data = df5['revenue_adj'], bins = 50, fill = True, ax = ax[1], alpha = 0.5, color = 'blue')
ax[0].legend()
fig.suptitle('Filling Zero Value to Mean')
plt.show()<jupyter_output>19792714.769019295
58902514.47627916<jupyter_text>Exploratory Data Analysis General(Compare every variable which can do.)<jupyter_code># ref: https://jehyunlee.github.io/2020/10/10/Python-DS-37-seaborn_matplotlib4/
g = sns.PairGrid(df6, diag_sharey = False, corner = True)

# diagonal
g.map_diag(sns.kdeplot, fill = True)
for i in range(3):
    g.axes[i][i].spines['left'].set_visible(True)
    g.axes[i][i].spines['top'].set_visible(True)
    g.axes[i][i].spines['right'].set_visible(True)
    
# lower
g.map_lower(sns.scatterplot, s = 30, edgecolor = 'w')
g.map_lower(sns.regplot, scatter = False, truncate = False, ci = False)
g.map_lower(sns.kdeplot, alpha = 0.5)

g.fig.suptitle("TMDB-Movie", y = 1.01, weight = "bold", fontsize = 'x-large')
g.fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>* Popularity has positive correlation budget and revenue!    - popular movie make many money    - and spend lots of money* If the movie company want to make lots of money, then they must spend lots of money!<jupyter_code>pick_keywords = df6.explode('keywords')['keywords'].value_counts().head(3).keys()

g = sns.PairGrid(df6.explode('keywords').query('keywords in {}'.format(list(pick_keywords))), hue = 'keywords', diag_sharey = False, corner = True)

# diagonal
g.map_diag(sns.kdeplot, fill = True)
for i in range(3):
    g.axes[i][i].spines['left'].set_visible(True)
    g.axes[i][i].spines['top'].set_visible(True)
    g.axes[i][i].spines['right'].set_visible(True)
    
# lower
# g.map_lower(sns.scatterplot, s = 30, edgecolor = 'w')
g.map_lower(sns.regplot, scatter = False, truncate = False, ci = False)
g.map_lower(sns.kdeplot, alpha = 0.5)

handles = g._legend_data.values()
labels = g._legend_data.keys()

# Insert legend
g.axes[1][0].legend(handles=handles, labels=labels,
                    bbox_to_anchor=(3.45, 1),
                    fontsize="large", frameon=False
                   )

g.fig.suptitle("TMDB-Movie Group by Top 3 Keywords, ", y = 1.01, weight = "bold", fontsize = 'x-large')
g.fig.tight_layout()
plt.show()<jupyter_output>D:\Sangwons_Room\00_SoftWares\Anaconda\envs\torch\lib\site-packages\ipykernel_launcher.py:27: UserWarning: Tight layout not applied. tight_layout cannot make axes width small enough to accommodate all axes decorations<jupyter_text>* Independent films can't make money, but you don't even have to spend money* The movies, which is based on novel, are very efficient(these are the highest revenue per budget on investment.)* Films directed by women have the highest profits compared to their popularity.<jupyter_code>pick_genres = df6.explode('genres')['genres'].value_counts().head(3).keys()

g = sns.PairGrid(df6.explode('genres').query('genres in {}'.format(list(pick_genres))), hue = 'genres', diag_sharey = False, corner = True)

# diagonal
g.map_diag(sns.kdeplot, fill = True)
for i in range(3):
    g.axes[i][i].spines['left'].set_visible(True)
    g.axes[i][i].spines['top'].set_visible(True)
    g.axes[i][i].spines['right'].set_visible(True)
    
# lower
# g.map_lower(sns.scatterplot, s = 30, edgecolor = 'w')
g.map_lower(sns.regplot, scatter = False, truncate = False, ci = False)
g.map_lower(sns.kdeplot, alpha = 0.5)

handles = g._legend_data.values()
labels = g._legend_data.keys()

# Insert legend
g.axes[1][0].legend(handles=handles, labels=labels,
                    bbox_to_anchor=(3.45, 1),
                    fontsize="large", frameon=False
                   )

g.fig.suptitle("TMDB-Movie Group by Top 3 Genres, ", y = 1.01, weight = "bold", fontsize = 'x-large')
g.fig.tight_layout()
plt.show()<jupyter_output>D:\Sangwons_Room\00_SoftWares\Anaconda\envs\torch\lib\site-packages\ipykernel_launcher.py:27: UserWarning: Tight layout not applied. tight_layout cannot make axes width small enough to accommodate all axes decorations<jupyter_text>* The revenue compared to the budget for each genres is similar  Research Question 1: How has the profitability of making films changed over time?<jupyter_code>df7 = pd.DataFrame(df6)
df7 = df7.sort_values(by = 'release_date')
df7['profit'] = df7['revenue_adj'] - df7['budget_adj']

fig = plt.figure(figsize = (12, 12))
spec = fig.add_gridspec(3, 3)

ax = {}
ax[0] = fig.add_subplot(spec[0,:])
ax[1] = fig.add_subplot(spec[1,:])
ax[2] = fig.add_subplot(spec[2,0])
ax[3] = fig.add_subplot(spec[2,1])
ax[4] = fig.add_subplot(spec[2,2])

    
df7.groupby('release_year').sum()[['budget_adj', 'revenue_adj', 'profit']].plot(kind = 'line', alpha = 0.8, ax = ax[0])
df7.groupby('release_year').mean()[['budget_adj', 'revenue_adj', 'profit']].plot(kind = 'line', alpha = 0.8, ax = ax[1])
df7.groupby('release_year').max()['budget_adj'].plot(kind = 'kde', alpha = 1, ax = ax[2])
df7.groupby('release_year').max()['revenue_adj'].plot(kind = 'kde', alpha = 1, ax = ax[3])
df7.groupby('release_year').max()['profit'].plot(kind = 'kde', alpha = 1, ax = ax[4])

ax[0].set_title('Sum of Total Profitability')
ax[1].set_title('Mean of Total Profitability')
ax[2].set_title('Histogram of Total Budget')
ax[3].set_title('Histogram of Total Revenue')
ax[4].set_title('Histogram of Total Profit')
ax[0].set_ylabel('Dolar(?)')
ax[1].set_ylabel('Dolar(?)')

ax[0].set_xlabel("")
ax[1].set_xlabel("")
ax[2].set_xlabel("Dolar(?)")
ax[3].set_xlabel("Dolar(?)")
ax[4].set_xlabel("Dolar(?)")
ax[3].set_ylabel("")
ax[4].set_ylabel("")

plt.show()<jupyter_output><empty_output><jupyter_text>* Movie industries are increasing since 1960s* Mean of budget, revenue, and profit is barely increase* Most movie budgets, revenues, and profits are as low as around $1e8.  Research Question 2: When do movies of each genre or keyword make the best profits?<jupyter_code>df8 = pd.DataFrame(df7)
pick_keywords = df8.explode('keywords')['keywords'].value_counts().head(5).keys()
pick_genres = df8.explode('genres')['genres'].value_counts().head(5).keys()

df8['month'] = df8['release_date'].dt.month
df_k = df8.explode('keywords')
df_k = df_k.query('keywords in {}'.format(list(pick_keywords)))
df_g = df8.explode('genres')
df_g = df_g.query('genres in {}'.format(list(pick_genres)))

fig, ax = plt.subplots(2,1,figsize = (16, 15))
sns.barplot(x = 'month', y = 'profit', data = df_k.groupby(['month', 'keywords']).mean().reset_index(), hue = 'keywords', ax = ax[0])
sns.barplot(x = 'month', y = 'profit', data = df_g.groupby(['month', 'genres']).mean().reset_index(), hue = 'genres', ax = ax[1])

ax[0].set_title('Monthly Trend of Movie Keyword')
ax[1].set_title('Monthly Trend of Movie Genre')

plt.show()<jupyter_output><empty_output><jupyter_text>* Films based on novels have high profits in February, March, July, and December.* Sports movies generate the most profits in November.* For the rest of the movies, there is no clear change from month to month.* Action movies make the biggest profit in May* Thriller movies make the biggest profit in June  ConclusionIn the EDA section, I want to see how much a movie's genre or keyword influences the movie's revenue or profit. First, I check correlation of all variables of interest. It is confirmed that the keyword or genre of the movie influenced the correlation of each variable. Second, I want to answer the first question, "How has the profitability of making films changed over time?". Most of the movies barely increase profits, but a handful of movies made huge amounts of money, and the amount is growing. Third, these keyword and genre are influenced by month? That is my question2, "When do movies of each genre or keyword make the best profits?". I can show you monthly trend of movies profit by keywords and genres. For some genres or keywords, monthly trends aren't clearly visible. Among the keywords, however, novel-based movies make remarkable profits in February, March, July and December, and sports keyword movies showed strong performance in November. In the genre of each movie, action make a lot of money in May, June, 7, and November, and the thriller genre show a strong tendency to profit in June and July. LimitationsMost of the data is not a lot of null data left after deleting unnecessary data. However, budget and revenue often include "0"(that is not null or nan), which have a big impact on the analysis results. In this analysis, the "0" value is simply corrected through the average of the whole value, but if the actual value cannot be found, an estimate value considering the director, genre, actor, etc. is needed. In addition, in order to infer a level that can be analyzed because there are a large amount of keywords or genres of movies, the analysis is conducted by simply selecting the five with the largest number of keywords or genres. For reliable analysis, analysis on each keyword and genre should be preceded.<jupyter_code>from subprocess import call
call(['python', '-m', 'nbconvert', 'Investigate_a_Dataset.ipynb'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13490980748825224, 'fraction_numerical': 0.02554191299075337, 'mean_word_length': 4.812775330396476, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '443874', 'license': 'MIT', 'n_tokens_mistral': 4590, 'n_tokens_neox': 4338, 'n_words': 1326, 'path': 'Knowledge/Python/210218 - [UDACITY]Introduction to Data Analysis/Investigate_a_Dataset.ipynb', 'repo_name': 'leeh8911/BeSuperRepo', 'size': '13194'}