<jupyter_start><jupyter_text>Phase 1 Project Technical NotebookAuthors: <NAME> and <NAME><jupyter_code>import seaborn as sns
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Rotten Tomatoes Data - Business Recommendation 1: High Critic Ratings Are Not Correlated With Box Office Success  Importing Rotten Tomatoes Files<jupyter_code>df_rt1 = pd.read_csv('zippedData/rt.movie_info.tsv.gz', sep='\t')
df_rt2 = pd.read_csv('zippedData/rt.reviews.tsv.gz', sep='\t', encoding='iso8859_1')
df_rt1.info()
df_rt2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 54432 entries, 0 to 54431
Data columns (total 8 columns):
 #   Column      Non-Null Count  Dtype 
---  ------      --------------  ----- 
 0   id          54432 non-null  int64 
 1   review      48869 non-null  object
 2   rating      40915 non-null  object
 3   fresh       54432 non-null  object
 4   critic      51710 non-null  object
 5   top_critic  54432 non-null  int64 
 6   publisher   54123 non-null  object
 7   date        54432 non-null  object
dtypes: int64(2), object(6)
memory usage: 3.3+ MB<jupyter_text>The two rotten tomatoes datasets share a common attribute: 'id", so we can merge them using this column  Data Cleaning:<jupyter_code># Merges the two Rotten Tomatoes dataframes on their shared column: 'id'
df_rt = df_rt1.merge(df_rt2, how='inner', on='id')
import string
df_rt['box_office'] = df_rt['box_office'].str.replace('[{}]'.format(string.punctuation), '')
# Removes the commas from the box office column
df_rt['box_office'] = df_rt['box_office'].apply(lambda x: float(x))
# Changes the box office column from strings to floats
df_rt['runtime'] = df_rt['runtime'].str.strip('minutes')
# Strips the 'minutes' from the runtime column so that we can convert to an int 
df_rt['theater_date'] = pd.to_datetime(df_rt['theater_date'], )
# Changes release date from string to datetime 
df_rt.dropna(subset=['rating_y'], inplace=True)
# drops null values in the rating column
df_rt['rating_y'][0:20]<jupyter_output><empty_output><jupyter_text>Based on the output above, the rating column is a mess. Some of the ratings are out of 5, some are out of 10, some are out of 4 and some are letter grades. If we want to use this column, we need to figure out a way to standardize it.<jupyter_code>def standardize(rating):
    if rating.endswith('5') and '/' in rating and '.' in rating:
        return (float(rating[0:3]) / float(rating[-1])) * 100
    elif rating.endswith('10') and '/' in rating and '.' in rating:
        return (float(rating[0:3]) / float(rating[-2:])) * 100
    elif rating.endswith('5') and '/' in rating:
        return (float(rating[0]) / float(rating[-1])) * 100
    elif rating.endswith('10') and '/' in rating and len(rating) < 5:
        return (float(rating[0]) / float(rating[-2:])) * 100
    elif rating.endswith('10') and '/' in rating and len(rating) == 5:
        return (float(rating[0:2]) / float(rating[-2:])) * 100
    elif rating.endswith('4') and '/' in rating and '.' in rating:
        return (float(rating[0:3]) / float(rating[-1])) * 100
    elif rating.endswith('4') and '/' in rating:
        return (float(rating[0]) / float(rating[-1])) * 100
    elif rating.endswith('5') and '/' in rating:
        return (float(rating[0]) / float(rating[-1])) * 100
    elif rating == 'A+':
        return 98
    elif rating == 'A':
        return 95
    elif rating == 'A-':
        return 92
    elif rating == 'B+':
        return 88
    elif rating == 'B':
        return 85
    elif rating == 'B-':
        return 82
    elif rating == 'C+':
        return 78
    elif rating == 'C':
        return 75
    elif rating == 'C-':
        return 72
    elif rating == 'D+': 
        return 68
    elif rating == 'D':
        return 65
    elif rating == 'D-':
        return 62
print(standardize('10/10'))
print(standardize('4.5/5'))
print(standardize('A+'))
print(standardize('4/4'))
print(standardize('3.0/4'))<jupyter_output>100.0
90.0
98
100.0
75.0<jupyter_text>The function above takes in a rating and returns an integer from 0 to 100. Because the majority of data points in the 'ratings_y' column are either out of 5, 10, or in letter grade format this function will give us standardized ratings for a majority of the entries.<jupyter_code>ratings = list(df_rt['rating_y'].value_counts().index[0:])
len(ratings)
# creates a list of all unique ratings so that we can test our function
standardized_list = []
for rating in ratings:
    new_rating = standardize(rating)
    standardized_list.append(new_rating)
    
none_count = []
for item in standardized_list:
    if item == None:
        none_count.append(item)

1 - (len(none_count)) / len(ratings)<jupyter_output><empty_output><jupyter_text>The above tells us that our function was able to capture ~60% of the unique review types. Although 60% doesn't sound great, the majority of reviews are in the common formats featured above so we won't lose too much data when we implement the function<jupyter_code>df_rt['rating_y'] = df_rt['rating_y'].apply(standardize)
df_rt.dropna(subset=['rating_y'], inplace=True)
df_rt<jupyter_output><empty_output><jupyter_text>Rotten tomatoes uses a "Tomatometer" to rate movies. If a critic likes the movie, they designate "fresh," if they don't enjoy it they desginate it "rotten" The tomatometer represents the percentage of fresh designations. Using the fresh column from the dataset, we can create a tomatometer for each movie.<jupyter_code>def fresh(rating):
    if rating == 'fresh':
        return 1
    elif rating == 'rotten':
        return 0<jupyter_output><empty_output><jupyter_text>This function returns a rating of 1 if the review is fresh and 0 if it is rotten. We can then use this to come up with  tomatometer for each movie - it will be the mean of all rotten/fresh values for a speficied movie<jupyter_code>df_rt['fresh'] = df_rt['fresh'].apply(fresh)
# applies the function to the fresh column
df_rt['currency'].value_counts()<jupyter_output><empty_output><jupyter_text>This shows us that all of the currency is in dollars so we don't have to worry about converting currency<jupyter_code>df_rt['rating_y'] = df_rt.groupby('id')['rating_y'].transform(
    lambda grp: np.mean(grp)
)
df_rt['fresh'] = df_rt.groupby('id')['fresh'].transform(
    lambda grp: np.mean(grp)
)
df_rt<jupyter_output><empty_output><jupyter_text>The above code first groups the data by indivual movie ('using the 'id' column) and then replaces the individual 'rating' the mean rating for that movie. The same process is repeated for the fresh column. Looking at the df above, we can now see that all values in the "fresh" and "rating" columns are the same for each individual movie (movies with the same value in the 'id' column)<jupyter_code>df_rt = df_rt.drop_duplicates(subset='id')
# This code drops duplicates in the 'id' column so now we only have one row for each movie<jupyter_output><empty_output><jupyter_text>Ideally, if the standardized rating function worked well, the 'fresh' column and 'rating_y' column will be correlated -- let's double check:<jupyter_code>np.corrcoef(df_rt['rating_y'], df_rt['fresh'])[1][0]<jupyter_output><empty_output><jupyter_text>That's a pretty strong correlation!   Business Understanding Let's use our two rating columns to see if there is any correlation between positive ratings and high box office earnings<jupyter_code>df_rt = df_rt.dropna(subset=['box_office'])
df_rt['box_office'] = df_rt['box_office'].apply(lambda x: x/1000000)
# Drops NA values in the box office column and then converts from dollars to $mm so it is easier to digest
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,4))
x1 = df_rt['rating_y']
x2 = df_rt['fresh']
y = df_rt['box_office']
ax1.scatter(x1,y);
ax2.scatter(x2,y);

ax1.set_title('Standardized Rating vs. Box Office Performance');
ax1.set_xlabel('Standardized Score');
ax1.set_ylabel('Domestic Box Office earnings ($mm)');

ax2.set_title('Tomatometer vs. Box Office Performance');
ax2.set_xlabel('Tomatometer');
ax1.set_ylabel('Domestic Box Office earnings ($mm)');
plt.savefig('Visualizations/RT_ratings_box_correlation.PNG')<jupyter_output><empty_output><jupyter_text>It doesn't look like there is necessarily a strong relationship between ratings and box office performance but let's double check the correlation to be sure:<jupyter_code>rating_correlation = round(np.corrcoef(df_rt['rating_y'], df_rt['box_office'])[1][0],3)
tomatometer_correlation = round(np.corrcoef(df_rt['fresh'], df_rt['box_office'])[1][0],3)

print("The correlation between standardized rating and box office revenue is {}".format(rating_correlation))
print("The correlation between tomatometer and box office revenue is {}".format(tomatometer_correlation))<jupyter_output>The correlation between standardized rating and box office revenue is 0.101
The correlation between tomatometer and box office revenue is 0.07<jupyter_text>Based on this analysis we did not find a correlation between critic ratings and box office success. Therefore, for future analysis, we will put less focus on critic reception, and instead focus on other attributes that might lead to profitability. Additionally, for future analysis we will focus on datasets that contain box office revenue information to guide our insights. As a result, we did not focus on several IMDB datasets within the "Zipped Data" file folder because they only contained ratings information  IMDB Data - Business Recommendation 2: Identifying Genres, Directors, Actors and Plot Themes that Translate to Higher Gross Revenue<jupyter_code>df_imdb = pd.read_csv('zippedData/movie_metadata.csv')
# Imports IMDB Data
df_imdb.info()
df_imdb['gross'] = df_imdb['gross'].apply(lambda x: x/1000000)<jupyter_output><empty_output><jupyter_text>The gross revenue column ('gross')  is not very digestible because it is in dollars, the code above turns it into millions<jupyter_code>def split_genres(genre):
    return genre.split('|')[0]

df_imdb['genres'] = df_imdb['genres'].apply(split_genres)<jupyter_output><empty_output><jupyter_text>In the IMDB dataset the 'genres' column has multiple entries separated by "|". For example, the first entry is 'Action|Adventure|Fantasy|Sci-Fi'. The above cell creates a function that returns the first main genre listed (so for the first entry it would return "Action"<jupyter_code>df_imdb.at[3080, 'genres'] = 'Sci-Fi'<jupyter_output><empty_output><jupyter_text>The above cell reclassifies, "E.T" from 'Family' to 'Sci-Fi'. Previously, the mean box office revenue for the 'Family' genre was skewed very high because there were only 3 movies in that category including E.T., a huge commercial success. Because there are more movies in the Sci-Fi category, E.T. will be less of on an outlier there.<jupyter_code>df_imdb['genres'].value_counts()<jupyter_output><empty_output><jupyter_text>Business Understanding<jupyter_code>df_imdb.groupby('genres')['gross'].mean().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>The above cell shows the mean gross revenue for movies in each genre and sorts them from highest to lowest. Based on the chart above, it is our recommendation that Microsoft should create a movie in either the Adventure, Animation, Action, Sci-Fi, Mystery or Fantasy Genre. Looking at the value counts, we can see that there are only 4 'Musical' movies in the data set, so, although movies in this genre appear to do well in the box office, we will not include this category moving forward because we do not have enough data points. Below we will try to make some visuals related to genre and then make recommendations of director, actor, and movie plot!<jupyter_code>x = list(df_imdb.groupby('genres')['gross'].mean().sort_values(ascending=False).index[0:10])
y= list(df_imdb.groupby('genres')['gross'].mean().sort_values(ascending=False).values[0:10])

fig, ax = plt.subplots(figsize=[12,8])
ax.bar(x,y,color = "dodgerblue");
ax.set_xlabel('Genre')
ax.set_ylabel('Average Domestic Gross ($mm)')
ax.set_title('Genres with Highest Average Domestic Gross');
plt.savefig('Visualizations/Genre_profitability.png')
df_genre_sub = df_imdb[df_imdb['genres'].isin(['Adventure', 'Action', 'Animation', 'Sci-Fi', 'Mystery', 'Fantasy'])]
# Creates a new DF that only contains movies in the specified genres
fig, ax = plt.subplots(figsize=(12,9))
ax = sns.boxplot(x='genres', y='gross', data=df_genre_sub);
ax.set_xlabel('Genre');
ax.set_ylabel('Domestic Box Office Gross');
ax.set_title('Most Profitable Genres');
plt.savefig('Visualizations/Genre_profitability_boxplot.png')
df_genre_sub['director_name'].value_counts()[0:100]<jupyter_output><empty_output><jupyter_text>The code above shows the 100 directors who have made the most movies in the genres we have specified. 100 is a bit of an arbitrary cutoff but we wanted to select directors that had experience. Even the 100th person on this list (<NAME>) has created 4 movies in the specified genres<jupyter_code>top_100_directors_for_genre = list(df_genre_sub['director_name'].value_counts().index[0:100])
# Creates a list containing names of the 100 directors who have made the most movies in the specified genre
df_director_sub = df_genre_sub[df_genre_sub['director_name'].isin(top_100_directors_for_genre)]
# Uses the list above to create a new dataframe with only entries for the above directors 
df_director_sub.groupby('director_name').mean()['gross'].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>The code above groups the directors and shows the average gross revenue for movies they've produced. Based on the output, we should target <NAME>, <NAME>, <NAME>, <NAME>, and <NAME> because they direct profitable movies<jupyter_code>x = df_director_sub.groupby('director_name').mean()['gross'].sort_values(ascending=False).index[0:10]
y = df_director_sub.groupby('director_name').mean()['gross'].sort_values(ascending=False).values[0:10]

fig, ax = plt.subplots(figsize=[12,8])
ax.bar(x,y, color='dodgerblue')
ax.set_xlabel('Director')
plt.xticks(rotation=45)
ax.set_ylabel('Average Domestic Gross per Movie ($mm)')
ax.set_title('Directors in Target Genres with Highest Average Domestic Gross');
plt.tight_layout() 
plt.savefig('Visualizations/Director_profitability.png')<jupyter_output><empty_output><jupyter_text>Data Understanding<jupyter_code>df_imdb.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5043 entries, 0 to 5042
Data columns (total 28 columns):
 #   Column                     Non-Null Count  Dtype  
---  ------                     --------------  -----  
 0   color                      5024 non-null   object 
 1   director_name              4939 non-null   object 
 2   num_critic_for_reviews     4993 non-null   float64
 3   duration                   5028 non-null   float64
 4   director_facebook_likes    4939 non-null   float64
 5   actor_3_facebook_likes     5020 non-null   float64
 6   actor_2_name               5030 non-null   object 
 7   actor_1_facebook_likes     5036 non-null   float64
 8   gross                      4159 non-null   float64
 9   genres                     5043 non-null   object 
 10  actor_1_name               5036 non-null   object 
 11  movie_title                5043 non-null   object 
 12  num_voted_users            5043 non-null   int64  
 13  cast_total_facebook_likes  5043 non-null   int64[...]<jupyter_text>Based on the output above, we can see that there are 3 seperate columns for actors<jupyter_code>df_2 = df_imdb.copy()
df_3 = df_imdb.copy()

# Creates two copies of the DF
df_imdb.drop(columns=['actor_2_name', 'actor_3_name'], inplace=True)
df_imdb.rename(index=str, columns={"actor_1_name": "actor"}, inplace=True)
# Deletes the actor_2 column and actor_3 column from the original DF and renames the actor_1 column 
df_2.drop(columns=['actor_1_name', 'actor_3_name'], inplace=True)
df_2.rename(index=str, columns={"actor_2_name": "actor"}, inplace=True)
# Deletes the actor_1 column and actor_3 column from the original DF and renames the actor_2 column 
df_3.drop(columns=['actor_1_name', 'actor_2_name'], inplace=True)
df_3.rename(index=str, columns={"actor_3_name": "actor"}, inplace=True)
# Deletes the actor_1 column and actor_3 column from the original DF and renames the actor_2 column 
actors = pd.concat([df_imdb,df_2, df_3])
actors.info()
assert actors['gross'][5043] == actors['gross'][0]
assert actors['actor'][5043] == actors['actor'][0]<jupyter_output><empty_output><jupyter_text>The above code was a little clunky but we now have a dataframe with all of the actors in one column. We combined 3 individual dataframes. The second dataframe contained actors that were previously in the actor_2 column while the 3rd dataframe contained actors that were previously in the actor_3 column. Each of these dataframes was 5,043 rows and each row represents a movie. In the concatanated dataframe, each movie has 3 row entries, one for each actor. The assert statement above shows us that the gross revenue for row 0 is the same as the gross revenue for row 5043 (because they are the same movie!). There is an error in the second assert statement because now they have seperate actors in the actor column  Business Understanding<jupyter_code>actors_genres_sub = actors[actors['genres'].isin(['Adventure', 'Action', 'Animation', 'Sci-Fi', 'Mystery', 'Fantasy'])]
# This code creates a new DF with only movies in the genres we highlighted before 
actors_genres_sub['actor'].value_counts()[0:200]<jupyter_output><empty_output><jupyter_text>Similar to the exercise we did with directors, the above output shows us the actors that have been in the most movies in the genres that we had selected as most profitable<jupyter_code>top_200_actors_in_genre = list(actors_genres_sub['actor'].value_counts().index[0:200])
# takes the above output and puts it into a list of actor names
actors_genres_sub = actors_genres_sub[actors_genres_sub['actor'].isin(top_200_actors_in_genre)]
# Filters out actors that aren't in the top 200 list we created 
actors_genres_sub.groupby('actor').mean()['gross'].sort_values(ascending=False)
# Sorts actors by the average profitability of the movies that they are in.

x = actors_genres_sub.groupby('actor').mean()['gross'].sort_values(ascending=False).index[0:10]
y = actors_genres_sub.groupby('actor').mean()['gross'].sort_values(ascending=False).values[0:10]

fig, ax = plt.subplots(figsize=[12,8])
ax.bar(x,y, color='dodgerblue')
ax.set_xlabel('Actor')
ax.set_ylabel('Average Domestic Gross per Movie ($mm)')
ax.set_title('Actors in Target Genres with Highest Average Domestic Gross');
plt.xticks(rotation=45)
plt.tight_layout()
plt.savefig('Visualizations/Actor_profitability.png')<jupyter_output><empty_output><jupyter_text>Based on the above, it looks like we should target <NAME>., <NAME>, <NAME>, <NAME> and <NAME> if we want a profitable movie!<jupyter_code>actors_genres_sub.groupby('actor').mean()['imdb_score'].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>The above sorts the top 200 actors in the specified genres by the average IMDB score for the movies they are in. Only a couple of the names in here appear in the chart above  Data Understanding<jupyter_code>df_keywords1 = pd.read_csv('zippedData/movie_metadata.csv')
df_keywords1['plot_keywords'][0]<jupyter_output><empty_output><jupyter_text>The dataset has a column for keywords in the plot (shown above), but similar to genre it has more than one keyword per movie and the keywords are divided by a "|".<jupyter_code>def split_keyword1(keywords):
      return keywords.split('|')[0]

def split_keyword2(keywords):
        if len(keywords.split('|')) > 1:
            return keywords.split('|')[1]
        else:
            return "No second keyword"

def split_keyword3(keywords):
        if len(keywords.split('|')) > 2:
            return keywords.split('|')[2]
        else:
            return "No third keyword"

def split_keyword4(keywords):
        if len(keywords.split('|')) > 3:
            return keywords.split('|')[3]
        else:
            return "No fourth keyword"

def split_keyword5(keywords):
        if len(keywords.split('|')) > 4:
            return keywords.split('|')[4]
        else:
            return "No fifth keyword"<jupyter_output><empty_output><jupyter_text>The code above is a little messy but it creates 5 different functions that each return one of the keywords in the column. The function "split_keyword1" returns the first keyword, the function "split_keyword2" returns the second keyword and so on. The if statement within the function exists in case there are titles with only 1 or 2 keywords. If this is the case, without the if statement there would be an error message if we attempted to run 'split_keyword5'. Instead, the function returns the string "No fifth keyword"<jupyter_code>df_keywords1.dropna(subset=['plot_keywords'], inplace=True)
df_keywords1['plot_keywords'] = df_keywords1['plot_keywords'].apply(split_keyword1)
df_keywords2 = pd.read_csv('zippedData/movie_metadata.csv')
df_keywords2.dropna(subset=['plot_keywords'], inplace=True)
df_keywords2['plot_keywords'] = df_keywords2['plot_keywords'].apply(split_keyword2)
df_keywords3 = pd.read_csv('zippedData/movie_metadata.csv')
df_keywords3.dropna(subset=['plot_keywords'], inplace=True)
df_keywords3['plot_keywords'] = df_keywords3['plot_keywords'].apply(split_keyword3)
df_keywords4 = pd.read_csv('zippedData/movie_metadata.csv')
df_keywords4.dropna(subset=['plot_keywords'], inplace=True)
df_keywords4['plot_keywords'] = df_keywords4['plot_keywords'].apply(split_keyword4)
df_keywords5 = pd.read_csv('zippedData/movie_metadata.csv')
df_keywords5.dropna(subset=['plot_keywords'], inplace=True)
df_keywords5['plot_keywords'] = df_keywords5['plot_keywords'].apply(split_keyword5)
df_keywords = pd.concat([df_keywords1,df_keywords2, df_keywords3, df_keywords4, df_keywords5])
df_keywords<jupyter_output><empty_output><jupyter_text>The above is a little repetitive. It creates 5 new dataframes and populated each one with one of the title keywords. dataframe 1 has the first keyword in the 'plot_keyword column', dataframe 2 has the second keyword in the 'plot_keyword' and so on.<jupyter_code>df_keywords = df_keywords[df_keywords['plot_keywords'] != "No second keyword"] 
df_keywords = df_keywords[df_keywords['plot_keywords'] != "No third keyword"] 
df_keywords = df_keywords[df_keywords['plot_keywords'] != "No fourth keyword"] 
df_keywords = df_keywords[df_keywords['plot_keywords'] != "No fifth keyword"] 

# The above eliminates all the instances in which the function didn't have a keyword<jupyter_output><empty_output><jupyter_text>Business Understanding<jupyter_code>x= list(df_keywords['plot_keywords'].value_counts().index[0:20])
y= list(df_keywords['plot_keywords'].value_counts().values[0:20])
fig, ax = plt.subplots(figsize=[12,8])
ax.bar(x,y, color='dodgerblue')
ax.set_xlabel('Plot Theme')
ax.set_ylabel('Movie Counts')
ax.set_title('Most Popular Movie Themes');
plt.xticks(rotation=90)
plt.tight_layout()
plt.savefig('Visualizations/Keyword_counts.png')<jupyter_output><empty_output><jupyter_text>Using our new dataframe of keywords, we created a list of the top 10 words and their associated counts above. The plot word that appears most often is "Love", awwwwwwwww! This list includes movies from all genres -- not just the genres that we had isolated when seeking directors and actors<jupyter_code>top_100_keywords = list(df_keywords['plot_keywords'].value_counts().index[0:100])
df_keyword_sub = df_keywords[df_keywords['plot_keywords'].isin(top_100_keywords)]
df_keyword_sub = df_keyword_sub.dropna(subset=['gross'])<jupyter_output><empty_output><jupyter_text>The above creates a list of 100 keywords that appear most frequently in the dataset and then filters out all movies that don't contain a keyword in this list and also filters out entries that don't have a value in the 'gross column'<jupyter_code>df_keyword_sub['gross'] = df_keyword_sub['gross'].apply(lambda x: x/1000000)
df_keyword_sub.groupby('plot_keywords').mean()['gross'].sort_values(ascending=False)
x = df_keyword_sub.groupby('plot_keywords').mean()['gross'].sort_values(ascending=False).index[0:20]
y = df_keyword_sub.groupby('plot_keywords').mean()['gross'].sort_values(ascending=False).values[0:20]
fig, ax = plt.subplots(figsize=[12,8])
ax.bar(x,y,color='dodgerblue')
ax.set_xlabel('Plot Theme')
ax.set_ylabel('Average Domestic Gross Revenue (in $mm)')
ax.set_title('Average Gross Revenue for Highest Grossing Plot Themes');
plt.xticks(rotation=90)
plt.tight_layout()
plt.savefig('Visualizations/Keyword_revenue.png')<jupyter_output><empty_output><jupyter_text>Above, we grouped movies by keywords (aka plot themes) and then calculated the average revenue of movies containing that theme. The top 10 themes associated with the highest revenue are listed above. Due to this output, we would recommend creating a movie that has one of these themes In this section we identified 6 genres that generally translate to higher profitability: Action, Adventure, Animation, Sci-Fi, Fantasy and Mystery. We then created a subset of the data featuring movies from these categories and identified the actors and directors that have been involved in the highest earning films. We also identified plot themes that appear in movies that are the most successful in the box office. Genres to target:  Action, Adventure, Animation, Sci-Fi, Fantasy, MysteryDirectors to target: <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME> <NAME>, <NAME>ameronActors to target: <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME> and <NAME>Plot Themes: superhero, princess, magic, battle, monster, assasin, terrorist, soldier, professor, king, future, spy, vampire, president, alien, island, CIA, scientist, revenge  Movie Randomizer<jupyter_code>import random

keyword_list = list(df_keyword_sub.groupby('plot_keywords').mean()['gross'].sort_values(ascending=False).index[0:50])
actor_list  = list(actors_genres_sub.groupby('actor').mean()['gross'].sort_values(ascending=False).index[0:50])
director_list = list(df_director_sub.groupby('director_name').mean()['gross'].sort_values(ascending=False).index[0:50])

key_themes_random = random.sample(keyword_list, 2)
actors_random = random.sample(actor_list, 2)
director_random = random.sample(director_list, 1)

print("Directed by " + director_random[0] + ", critics are raving about Microsoft Studio's first film. \
Anchored by fantastic performances by " + actors_random[0] +" and " + actors_random[1] \
+ ", the film explores important themes such as " + key_themes_random[0] + "(s) and " + key_themes_random[1] + "(s). Microsoft \
has set records at the box office with this film, and the future is bright for the new studio!")<jupyter_output>Directed by <NAME>, critics are raving about Microsoft Studio's first film. Anchored by fantastic performances by <NAME> and <NAME>, the film explores important themes such as based on novel(s) and robot(s). Microsoft has set records at the box office with this film, and the future is bright for the new studio!<jupyter_text>Based on our recommendations above, we decided to build a random movie generator that draws on the director, actor and plot theme lists we created. For the purpose of the randomizer, we broadened each list to include 50 elements rather than 10 or 20 that we had identified in the chart. Have fun cycling through!   The Numbers Data<jupyter_code>df = pd.read_csv('zippedData/tn.movie_budgets.csv.gz')
df.head()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5782 entries, 0 to 5781
Data columns (total 6 columns):
 #   Column             Non-Null Count  Dtype 
---  ------             --------------  ----- 
 0   id                 5782 non-null   int64 
 1   release_date       5782 non-null   object
 2   movie              5782 non-null   object
 3   production_budget  5782 non-null   object
 4   domestic_gross     5782 non-null   object
 5   worldwide_gross    5782 non-null   object
dtypes: int64(1), object(5)
memory usage: 271.2+ KB<jupyter_text>Data Cleaning Luckily, we have no missing values. However, we need the data columns for production_budget, domestic_gross, and worldwide_gross to be integers, rather than strings, in order to properly analyze them.<jupyter_code>#cleaning the production budget column, domestic_gross column, worldwide_gross column to be floats
df['prod_budg_fixed'] = df['production_budget'].str.replace('$', '').str.replace(',','').astype(float)
df['dom_gross_fixed'] = df['domestic_gross'].str.replace('$', '').str.replace(',','').astype(float)
df['world_gross_fixed'] = df['worldwide_gross'].str.replace('$', '').str.replace(',','').astype(float)
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Data AnalysisWith the cleaning out of the way, we have a column of floats for production budget called prod_budg_fixed, a column of floats for domestic gross called dom_gross_fixed, and a column of floats for worldwide gross called world_gross_fixed. We can now start exploring relationships between production budget and the gross revenue<jupyter_code>#Checking the Pearson Correlation Coefficient
df.corr(method='pearson')<jupyter_output><empty_output><jupyter_text>From here, we can see that there is a relatively high correlation between production budget and worldwide gross (0.748306), and an extremely high correlation between the domestic gross and worldwide gross, indicating that a movie doing well in domestic theaters will also do well internationally. Logical, but nice to know.<jupyter_code>#scatterplot of production budget vs worldwide gross
import matplotlib.pyplot as plt
from matplotlib.axis import Axis
import seaborn as sns
import numpy as np
import pylab as plb


#code for basic scatter plot
x = df['prod_budg_fixed']
y = df['world_gross_fixed']

fig, ax = plt.subplots(figsize=(16, 10))

#going for more of a 'heatmap' style plot for readability, so opacity is low
ax.scatter(x, y, alpha=0.5)
#domestic gross scatter plot
y2 = df['dom_gross_fixed']
ax.scatter(x, y2, alpha=0.5, c='pink')

ax.set_title('Production Budget vs. Gross Revenue')
ax.set_xlabel('Production Budget')
ax.set_ylabel('Worldwide Gross (Blue) and Domestic Gross (Orange)')
ax.set_xlim(xmin=-5000, xmax=(3*10**8))
ax.set_ylim(ymin=-5000, ymax=(2*10**9))
plt.legend(['Worldwide', 'Domestic'])


#finding the trend line
z = np.polyfit(x,y,1)
p = np.poly1d(z)
plb.plot(x, p(x), 'b--')
#Supposedly the equation for the line (need to look into this more to understand it)
print("y=%.6fx+(%.6f)"%(z[0],z[1]))

#second trend line
z2 = np.polyfit(x,y2,1)
p2 = np.poly1d(z2)
plb.plot(x, p2(x), 'r--')
print("y2=%.6fx+(%.6f)"%(z2[0],z2[1]))<jupyter_output>y=3.126943x+(-7285667.054592)
y2=1.119087x+(6523888.576660)<jupyter_text>Based on what we see of the scatterplot above, as well as the trend lines, there is a positive correlation for both, with the worldwide gross having a considerably higher slope (meaning higher returns (gross) on investment (production budget)). Releasing a film internationally generally gives better returns than releasing a film purely domestically. Taking a look at the trend line linear equations: worldwide gross: y=3.126943x+(-7285667.054592) domestic gross: y2=1.119087x+(6523888.576660)Using these equations, we can make extremely general estimates of gross domestic earnings based on our production budget using basic linear algebra. x = production budgety-intercept (worldwide gross) = -7285667.05y2-intercept (domestic gross) = 6523888.57x-intercept (breakeven point for each):    y (worldwide<jupyter_code>#Variance of each column
df['world_gross_fixed'].var()
df['dom_gross_fixed'].var()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10949731208148637, 'fraction_numerical': 0.026344745197899967, 'mean_word_length': 5.302754111353279, 'pattern_counts': {'":': 3, '<': 191, '<?xml version=': 0, '>': 194, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '517223', 'license': 'RSA-MD', 'n_tokens_mistral': 10551, 'n_tokens_neox': 9751, 'n_words': 3222, 'path': 'technical_notebook.ipynb', 'repo_name': 'KPMcDonough49/dsc-phase-1-project', 'size': '32091'}