<jupyter_start><jupyter_text>![example](images/director_shot.jpeg)  Recommendations for a New Movie Studio**Authors:** <NAME>, <NAME>, <NAME>, <NAME>***  OverviewMicrosoft wants to create a new original video content creation studio. We utilized data from IMDb and The Numbers; such as, Total Gross, ROI, Run Time, Genres, and Directors. Data was researched, using the Python library Pandas, to perform exploratory analysis into what attributes are currently dominating the film industry, and in order to provide Microsoft reccomendations we used basic data visualization libraries like Matplotlib, and Seaborn.  Business ProblemMicrosoft has expressed interest in going into the original video creation by creating a new movie studio. The Studio team has tasked us to explore which types of films are dominating the market, and translate them into actionable intel that can help the head of Microsofts new studio makean informed decision on which films they would like to start to produce.  Data UnderstandingDatasets were scraped from IMDb.com and The-Numbers.com, data from IMDb contains data regarding movie runtime, genre, and directors. The-Numbers dataset contains information relating to worldwide gross and production budget, which allowed us to calculate profit and return on investment(ROI). The dataset also included release dates, which allowed us to calculate the importance of season. We cleaned datasets to only contain information related the last ten years, to insure our reccomendations reflect the current market. Information from both datasets were joined to compare variables against total profit and ROI in order to produce meaningful recommendations.<jupyter_code># Import standard packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import calendar

%matplotlib inline
df_crew = pd.read_csv('Data/zippedData/imdb.title.crew.csv.gz')
df_name = pd.read_csv('Data/zippedData/imdb.name.basics.csv.gz')
df_title_basic = pd.read_csv('Data/zippedData/imdb.title.basics.csv.gz')
df_num = pd.read_csv('Data/zippedData/tn.movie_budgets.csv.gz')
df_crew.info()
df_name.info()
df_title_basic.info()
df_num.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5782 entries, 0 to 5781
Data columns (total 6 columns):
 #   Column             Non-Null Count  Dtype 
---  ------             --------------  ----- 
 0   id                 5782 non-null   int64 
 1   release_date       5782 non-null   object
 2   movie              5782 non-null   object
 3   production_budget  5782 non-null   object
 4   domestic_gross     5782 non-null   object
 5   worldwide_gross    5782 non-null   object
dtypes: int64(1), object(5)
memory usage: 271.2+ KB<jupyter_text>Film Crew Information This dataset contains the unique id number for the crew involved with making the corresponding movies, we used this dataset to obtain the names of film directors.<jupyter_code>df_crew.head()
df_crew['directors'].value_counts()<jupyter_output><empty_output><jupyter_text>Based on the intial look at the data, directors are seperated by a unique id name that we can cross reference with another database to pull their names, and the movies they worked on. We want to pull only movies with a single director, as they represent a majority of the data.  Director Name Information Using the information in this dataset, we can extract the name of the director from their unique id associated with them. We can now tie director name to the corresponding movie title's using both the previous and the next dataset (using tconst).<jupyter_code>df_name.head()<jupyter_output><empty_output><jupyter_text>Title Basics This is one of the main datasets that we are going to use to create suggestions, useful information like title, genre, and runtime are all available to us within<jupyter_code>df_title_basic.head()
df_title_basic['start_year'].value_counts()
# Most of the information is from the past 10 years, however there are some funky data points so we will still hard code a limit into the data.
df_title_basic['genres'].value_counts()
#Need to deal with multi-genre movies, and to trim down to the most relevant genres<jupyter_output><empty_output><jupyter_text>Using the 'tconst' column, we can tie in the other supplementary datasets to include in our analysis. We can start to form questions off of this dataset such as, 'How does runtime affect the rating/ROI' or 'do certain genres dominate the market more than others?'  The Numbers File Three of the most important datapoints come from this set, production budget, worldwide gross, and release date. Utlizing production budget and worldwide gross we can compute both net profit and return on investment (ROI). This dataset will be on main set that we merge others into. We will need to trim it down to only contain movies from 2010 and onwards (last 10 years).<jupyter_code>df_num.head()
df_num['worldwide_gross'].describe()
#All the number data are objects, we will need to convert them to int/float to provide more analysis<jupyter_output><empty_output><jupyter_text>Its a common idea that movies that come out during the summer and beginning of the holiday season perform better, using this data we can figure out how much it truly affects the total profitability.  Data PreparationInitial data preparation ranged between datasets. This included the removal of null values/incomplete entries, the transforming of certain data types into usable fields, and the trimming of data to only contain information from 2010-2021. It also included the creation of several new variables (profit, ROI) as well as the dropping of variables not used. This initial preparation and cleaning was done in order for us to merge all datasets into a final usable data frame from which we can draw meaniful conclusions/recommendations and produce our visuals.  Film Crew Cleaning<jupyter_code># Delete 'writers' 
df_crew.drop(['writers'], axis=1, inplace=True)

# Drop missing values in 'directors'
df_crew.dropna(axis=0, subset=['directors'], inplace=True)

# Drop duplicate in 'tconst', film id
df_crew.drop_duplicates(subset='tconst', inplace=True)

# Split directors into director1, director2, director3, director4 
# 'director2' has 2nd director and others (if more than 2)
df_crew[['director1', 'director2']] = df_crew['directors'].str.split(',', n=1,  expand =True)

# Drop 'directors' and 'director2' columns.
# Only keeping director1 for each movie
df_crew.drop(['directors', 'director2'], axis=1, inplace=True)
df_crew.head()<jupyter_output><empty_output><jupyter_text>Director Name Cleaning<jupyter_code># Keep only columns 'nconst', 'primary_name'
df_name = df_name[['nconst', 'primary_name' ]]

# Checking for any null values
df_name.isna().value_counts()
# Checking for any duplicate entries
df_name.duplicated().value_counts()
df_name.head()<jupyter_output><empty_output><jupyter_text>Title Basics and Genre Cleaning<jupyter_code># Dropping all null values from dataset, incomplete entries would create merge conflicts
df_title_basic.dropna(inplace = True)

# Subset data to only include year 2010-2021
df_title_2010 = df_title_basic[(df_title_basic["start_year"] >= 2010) & (df_title_basic["start_year"] <= 2021)] 

# Drop the primary_title column, we will use original_title to avoid as many potential movies with the same name
df_title_final = df_title_2010.drop("primary_title", axis=1)

# Checking for any duplicate entries
df_title_final.duplicated().value_counts()
df_title_final.head()<jupyter_output><empty_output><jupyter_text>The Numbers Cleaning<jupyter_code># Define function to transform money values (object) into usable numbers (float)
def data_cleaning_money(column_name):
    df_num[column_name] = df_num[column_name].str.replace('$', '').str.replace(',', '').astype(float)

# Tranform product_budget and worldwide_gross and drop domestic_gross and id columns
data_cleaning_money('worldwide_gross')
data_cleaning_money('production_budget')
df_num = df_num.drop(columns = 'id')

# Transforming release_date into datetime object for easier use
df_num['release_date'] = pd.to_datetime(df_num['release_date'])

# Cleaning data and removing any potential null (0.0) values from relevant columns. 
df_num = df_num[df_num['domestic_gross'] != 0.0]
df_num = df_num[df_num['worldwide_gross'] != 0.0]
df_num = df_num[df_num['production_budget'] != 0.0]

# Drop any rows where domestic_gross = worldwide_gross, indictating no value for foreign gross
df_num = df_num[df_num['domestic_gross'] != df_num['worldwide_gross']]

# Drop domestic gross once cleaning is done
df_num_cleaned = df_num.drop('domestic_gross', axis = 1)
# Create new variable 'profit'
df_num_cleaned['profit'] = (df_num_cleaned['worldwide_gross'] - df_num_cleaned['production_budget'])

# Using 'profit' variable, create new variable "ROI" (Return On Investment)
df_num_cleaned['ROI'] = (df_num_cleaned['profit']) / (df_num_cleaned['production_budget'])

#Using all the data that comes after January 1st, 2010
df_num_cleaned.drop(df_num_cleaned[df_num_cleaned['release_date'] < pd.Timestamp(2010, 1, 1)].index, inplace = True)

#Creating month column
df_num_cleaned['month'] = df_num_cleaned['release_date'].dt.month

#Creating month name column using calendar library
df_num_cleaned['month_name'] = df_num_cleaned['month'].apply(lambda x: calendar.month_name[x])
df_num_cleaned.drop(['release_date', 'month'], axis=1, inplace=True)
df_num_cleaned.head()<jupyter_output><empty_output><jupyter_text>Merging of datasets Consolidating our dataframes into one to perform our data visualizations/analysis<jupyter_code>#Merging the director ids with the director names
df_merge = df_crew.merge(df_name, how = 'inner', left_on = 'director1', right_on = 'nconst')

#dropping the irrelevant merge data
df_merge.drop(['director1', 'nconst'], axis=1, inplace=True)

#renaming director column
df_merge.rename(columns={'primary_name' : 'director_name'}, inplace=True)
df_merge.head()
#second merge, adding director to the df_title data frame
df_merge1 = df_merge.merge(df_title_final, how = 'inner', left_on = 'tconst', right_on = 'tconst')
#dropping irrelevant data
df_merge1.drop(['tconst', 'start_year'], axis=1, inplace=True)

df_merge1.head() 
#final merge, adding the merged datafram on the numbers file
df_merge_final = df_num_cleaned.merge(df_merge1, how = 'inner', left_on = 'movie', right_on = 'original_title')

#dropping similar column
df_merge_final.drop('original_title', axis=1, inplace=True)

#dropping duplicated title so money stats aren't inflated with bad data
df_merge_final.drop_duplicates(subset=['movie'], inplace=True)

#splitting the genres so we can use the data for visualizing
df_merge_final[['genre1', 'genre2']] = df_merge_final['genres'].str.split(',', n=1, expand=True)

#dropping duplicate data/irrelevant data
df_merge_final.drop(['genres', 'genre2'], axis=1, inplace=True)

df_merge_final.head()
horror = df_merge_final.filter('Items'='Horror')<jupyter_output><empty_output><jupyter_text>Data ModelingDescribe and justify the process for analyzing or modeling the data.***Questions to consider:* How did you analyze or model the data?* How did you iterate on your initial approach to make it better?* Why are these choices appropriate given the data and the business problem?***<jupyter_code>df_merge_final.plot.scatter('production_budget', 'ROI');
df_merge_final.plot.scatter('production_budget', 'profit');
df_merge_final.plot.scatter('runtime_minutes', 'profit');
df_merge_final.plot.scatter('runtime_minutes', 'ROI');
df_merge_final.plot.scatter('worldwide_gross', 'ROI');
df_merge_final.plot.scatter('month_name', 'ROI')
plt.xticks(rotation=70);
df_merge_final.plot.scatter('genre1', 'ROI')
plt.xticks(rotation=90);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08972614390202416, 'fraction_numerical': 0.012502126211940806, 'mean_word_length': 6.007747318235995, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '483571', 'license': 'MIT', 'n_tokens_mistral': 3609, 'n_tokens_neox': 3445, 'n_words': 1259, 'path': 'Data/Code/Colins-not-deleted-file.ipynb', 'repo_name': 'ZenBurrito/Phase-1Project', 'size': '11786'}