<jupyter_start><jupyter_text>Project: Investigate The Movie Database (TMDb) Table of ContentsIntroductionData WranglingExploratory Data AnalysisConclusions  Introduction> This data set contains information about 10,000 movies collected from The Movie Database (TMDb).This project is aiming at investigating this data set to define factors that influence user ratings and revenue.>>Data was downloaded from >https://d17h27t6h515a5.cloudfront.net/topher/2017/October/59dd1c4c_tmdb-movies/tmdb-movies.csv.  uploading the selected file to the workspace<jupyter_code># Use this cell to set up import statements for all of the packages that you
#   plan to use.

# Import needed packages
import pandas as pd
import numpy as np
% matplotlib inline
import matplotlib.pyplot as plt


# Remember to include a 'magic word' so that your visualizations are plotted
#   inline with the notebook. See this page for more:
#   http://ipython.readthedocs.io/en/stable/interactive/magics.html

# Read Data File
df = pd.read_csv("tmdb-movies.csv")<jupyter_output><empty_output><jupyter_text>Data Wrangling> **Tip**: In this section of the report, you will load in the data, check for cleanliness, and then trim and clean your dataset for analysis. Make sure that you document your steps carefully and justify your cleaning decisions. General Properties<jupyter_code># Load your data and print out a few lines. Perform operations to inspect data
#   types and look for instances of missing or possibly errant data.
#Investigate the data file
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 21 columns):
id                      10866 non-null int64
imdb_id                 10856 non-null object
popularity              10866 non-null float64
budget                  10866 non-null int64
revenue                 10866 non-null int64
original_title          10866 non-null object
cast                    10790 non-null object
homepage                2936 non-null object
director                10822 non-null object
tagline                 8042 non-null object
keywords                9373 non-null object
overview                10862 non-null object
runtime                 10866 non-null int64
genres                  10843 non-null object
production_companies    9836 non-null object
release_date            10866 non-null object
vote_count              10866 non-null int64
vote_average            10866 non-null float64
release_year            10866 non-null int64
budget_adj              1[...]<jupyter_text>> I wanted to know if all data types are formed the way I need and if there are any columns that I might not use in my analysis.<jupyter_code>df.head(10)<jupyter_output><empty_output><jupyter_text>> I wanted to have a look on the data before performing any operations.>> I noted that the cells in the Cast columns are not arranged alphabetically. I have to assume that the order of the stars is arranged based on their importance to the box-office.Also some movies have more than one director. However and unlike with the actors, usually the credit is equally distributed among the directors of the same movie. So I am not planning to split this column.> Basically, I need to split the Cast and genres columns.So I can analyze data based on star name and genres. > **Tip**: You should _not_ perform too many operations in each cell. Create cells freely to explore your data. One option that you can take with this project is to do a lot of explorations in an initial notebook. These don't have to be organized, but make sure you use enough comments to understand the purpose of each code cell. Then, after you're done with your analysis, create a duplicate notebook where you will trim the excess and organize your steps so that you have a flowing, cohesive report.> **Tip**: Make sure that you keep your reader informed on the steps that you are taking in your investigation. Follow every code cell, or every set of related code cells, with a markdown cell to describe to the reader what was found in the preceding cell(s). Try to make it so that the reader can then understand what they will be seeing in the following cell(s). Data Cleaning: deleting duplicates and not needed columns<jupyter_code>sum(df.duplicated())<jupyter_output><empty_output><jupyter_text>> Finding the duplicates in order to delete them<jupyter_code># deleting columns that I am not going to use in my analysis. 
del df['homepage']
del df['imdb_id']
del df['tagline']
del df['keywords']<jupyter_output><empty_output><jupyter_text>>Reducing the data size will reduce the time needed to run my queries. Accordingly, I decided to skip the columns I am planning not to use. The deleted columns will not influence user rating or revenue.<jupyter_code>df.info()
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Identifying missing data<jupyter_code># Since all the null vlues are in object-type columns, 
# I will fill Null cells with 'Unknown'
df = df.fillna('Unknown')<jupyter_output><empty_output><jupyter_text>> Replacing missing data with the word 'Unknown'<jupyter_code>df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Making sure that there are no missing data in the file<jupyter_code>#Drop duplicates
df.drop_duplicates(inplace=True)
sum(df.duplicated())<jupyter_output><empty_output><jupyter_text>Duplicates deleted<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 10865 entries, 0 to 10865
Data columns (total 17 columns):
id                      10865 non-null int64
popularity              10865 non-null float64
budget                  10865 non-null int64
revenue                 10865 non-null int64
original_title          10865 non-null object
cast                    10865 non-null object
director                10865 non-null object
overview                10865 non-null object
runtime                 10865 non-null int64
genres                  10865 non-null object
production_companies    10865 non-null object
release_date            10865 non-null object
vote_count              10865 non-null int64
vote_average            10865 non-null float64
release_year            10865 non-null int64
budget_adj              10865 non-null float64
revenue_adj             10865 non-null float64
dtypes: float64(4), int64(6), object(7)
memory usage: 1.5+ MB<jupyter_text>Data is clean and proceeding with splitting the Cast column<jupyter_code>df.head(4)
#Splitting 'cast' column

dfCast= df['cast'].str.split('|', expand=True)

#source: https://stackoverflow.com/questions/14745022/how-to-split-a-column-into-two-columns

dfCast.head(5)
#rename columns
dfCast = dfCast.rename(columns={0: '1st_Star', 1: '2nd_Star', 2:'3rd_Star', 3: '4th_Star', 4: '5th_Star'})
dfCast.head(5)
#Appending new cast columns
dffinal = pd.concat([dfCast, df], axis=1, join_axes=[df.index])
#Splitting 'genres' column

dfGenres= df['genres'].str.split('|', expand=True)
dfGenres.head(5)
dfGenres = dfGenres.rename(columns={0: '1st_Genre', 1: '2nd_Genre', 2:'3rd_Genre', 3: '4th_Genre', 4: '5th_Genre'})
dfGenres.head(5)
#Appending new genre columns
dffinal = pd.concat([dfGenres, dffinal], axis=1, join_axes=[dffinal.index])
dffinal.head(5)
dffinal.info()
#Filling the missing data
dffinal = dffinal.fillna('Unknown')
dffinal.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 10865 entries, 0 to 10865
Data columns (total 27 columns):
1st_Genre               10865 non-null object
2nd_Genre               10865 non-null object
3rd_Genre               10865 non-null object
4th_Genre               10865 non-null object
5th_Genre               10865 non-null object
1st_Star                10865 non-null object
2nd_Star                10865 non-null object
3rd_Star                10865 non-null object
4th_Star                10865 non-null object
5th_Star                10865 non-null object
id                      10865 non-null int64
popularity              10865 non-null float64
budget                  10865 non-null int64
revenue                 10865 non-null int64
original_title          10865 non-null object
cast                    10865 non-null object
director                10865 non-null object
overview                10865 non-null object
runtime                 10865 non-null int64
genres              [...]<jupyter_text>Exploratory Data Analysis> **Tip**: Now that you've trimmed and cleaned your data, you're ready to move on to exploration. Compute statistics and create visualizations with the goal of addressing the research questions that you posed in the Introduction section. It is recommended that you be systematic with your approach. Look at one variable at a time, and then follow it up by looking at relationships between variables. Research Question 1: Is the revenue related to the budget? > Use this, and more code cells, to explore your data. Don't forget to add>   Markdown cells to document your observations and findings.<jupyter_code>plt.rcParams['figure.figsize']=(10,8)
dffinal.plot(x= 'budget', y='revenue', kind='scatter');
plt.title('Scatter Plot showing the relationship between budget and revenue')
plt.xlabel('Budget (100 Million Dollars)')
plt.ylabel('Revenue (Billion Dollars)')
plt.show()<jupyter_output><empty_output><jupyter_text>The scatter plot indicates that to some extent as the budget increases the revenue also increases. However, this conclusion might be premature because there are many points where the budget was extremely high yet the revenue did not come parallel to the expectation.To have a better view about the relationship between budget and revenue I will need to go deeper in understanding the budget parameter.<jupyter_code>dffinal.describe()
dffinal['budget'].describe()
plt.rcParams['figure.figsize']=(10,6)

dffinal['budget'].plot.box(vert = False)

plt.title('Box Plot summarizing Budget')
plt.xlabel('Budget (100 Million Dollars)')

plt.show()
plt.rcParams['figure.figsize']=(10,6)
dffinal['budget'].plot.hist()

plt.title('Histogram summarizing Budget')
plt.xlabel('Budget (100 Million Dollars)')

plt.show()
dffinal['budget'].head(10)<jupyter_output><empty_output><jupyter_text>The mean and the very high standard deviaion in addition to the histogram and the boxplot indicate that the budget might not be normaly distributed. For better intuition about the role of the budget in predecting the revenue I will split the data to low budget movies (<mean budget) and high budget movies (below mean budget) and high budget movies (costed an above mean budget).<jupyter_code># get the mean budget (Mean = dffinal.budget.mean())

# Split: Low budget movies
df_LowBud = dffinal[dffinal.budget < (dffinal.budget.mean())]

plt.rcParams['figure.figsize']=(12,8)

df_LowBud.plot(x= 'budget', y='revenue', kind='scatter');

plt.title('Scatter Plot showing the relationship between budget and revenue for low budget movies')
plt.xlabel('Budget (100 Million Dollars)')
plt.ylabel('Revenue (Billion Dollars)')
plt.show()

# Split: Low budget movies
df_HighBud = dffinal[dffinal.budget > (dffinal.budget.mean())]
plt.rcParams['figure.figsize']=(12,8)
df_HighBud.plot(x= 'budget', y='revenue', kind='scatter');

plt.title('Scatter Plot showing the relationship between budget and revenue for high budget movies')
plt.xlabel('Budget (100 Million Dollars)')
plt.ylabel('Revenue (Billion Dollars)')
plt.show()<jupyter_output><empty_output><jupyter_text>Now it is more obvious that as the budget increases the revenue increases. This effect is more obvious for movies with relatively high budgets (> mean budgets in the market).  Research Question 2: Which first movie star was associated with the highest popularity and revenue? Group by 1st starI tried to run the query  > dffinal.groupby('1st_Star').describe().popularityThis query took so much time. So it is better to split the file based on popularity. I will only include the highest percentile (> 75%).<jupyter_code>dffinal.describe().popularity
df_topPopular = dffinal[dffinal.popularity > 0.713857 ]
PopularStar = df_topPopular.groupby('1st_Star').describe().popularity
PopularStar.head()
PopularStar.sort_values(['mean', 'max', 'count']).tail(5)
#Displaying top popular movies
dffinal.sort_values(['popularity']).tail(5)
# Displaying hidden columns
pd.set_option('display.max_columns', 50)

#source: http://songhuiming.github.io/pages/2017/04/02/jupyter-and-pandas-display/
#Displaying top popular movies
dffinal.sort_values(['popularity']).tail(5)<jupyter_output><empty_output><jupyter_text>> <NAME> was the main star in the most popular movies during the study period. He participated in two movies (Jurassic World & Guardians of the Galaxy) with a budget of 150 and 170 Million Dollars respectively. Both movies were ranked in the top 5 popular movies. > <NAME>, <NAME>, <NAME> and <NAME> are in the top 5 popular main movie star list. They all made only one movie except <NAME>ill who made three movies. > The last table also shows that action, adventure and science fiction are the general genres in the top popular movies.However, to find out which movie genre was the more frequent in the data, then a more complex query must be written.   Research Question 3: which movie genres were more frequent?<jupyter_code>dffinal['1st_Genre'].unique()
Genres1 = dffinal['1st_Genre']
Genres2 = dffinal['2nd_Genre']
Genres3 = dffinal['3rd_Genre']
Genres4 = dffinal['4th_Genre']
Genres5 = dffinal['5th_Genre']


Genres = Genres1.append(Genres2)
Genres = Genres.append(Genres3)
Genres = Genres.append(Genres4)
Genres = Genres.append(Genres5)



from collections import Counter
#source: https://stackoverflow.com/questions/2600191/how-to-count-the-occurrences-of-a-list-item

x = Counter(Genres)
print(x)
#Sorting the file
del x['Unknown']
x.most_common()

#Source: https://stackoverflow.com/questions/20950650/how-to-sort-counter-by-value-python
z = x.keys()
y = x.values()

plt.rcParams['figure.figsize']=(25,6)

plt.bar(z,y, align='center', );
plt.title('Bar chart showing the frequencies of each movie genre through the years')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08278990643606464, 'fraction_numerical': 0.038417918911256024, 'mean_word_length': 4.08982683982684, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 97, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '308639', 'license': 'MIT', 'n_tokens_mistral': 4525, 'n_tokens_neox': 4039, 'n_words': 1550, 'path': 'Project_investigate-a-dataset_EmadFawaz.ipynb', 'repo_name': 'EmadFawaz/Investigate_the_Movie_Database', 'size': '14137'}