<jupyter_start><jupyter_text>Project 2: Investigate the dataset TMDb movie data Table of ContentsIntroductionData WranglingExploratory Data AnalysisConclusions  Introduction To create this analysis I choose the dataset called 'TMDb movie data'. This data set contains information about 10,000 movies collected from The Movie Database (TMDb), including user ratings and revenue. The questions that I will answer on this report are the next ones:- Which genres are most popular from year to year?- What kinds of properties are associated with movies that have high revenues?  Data Wrangling General Properties First of all I will load the libraries that I will use in this analysis.<jupyter_code># Import the libraries
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
% matplotlib inline<jupyter_output><empty_output><jupyter_text>Next I will load the dataset from the CSV file called 'tmdb-movies.csv'.<jupyter_code># Read the data
df = pd.read_csv('tmdb-movies.csv')

#Show the first 5 rows
df.head()
#Shape of the dataset
df.shape<jupyter_output><empty_output><jupyter_text>As we can see from the previous preview, this dataset contains in total 21 columns and 10866 rows.Now I will check the number of NA and datatype of each feature.<jupyter_code>#Check the number of NA and data type of each feature
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 21 columns):
id                      10866 non-null int64
imdb_id                 10856 non-null object
popularity              10866 non-null float64
budget                  10866 non-null int64
revenue                 10866 non-null int64
original_title          10866 non-null object
cast                    10790 non-null object
homepage                2936 non-null object
director                10822 non-null object
tagline                 8042 non-null object
keywords                9373 non-null object
overview                10862 non-null object
runtime                 10866 non-null int64
genres                  10843 non-null object
production_companies    9836 non-null object
release_date            10866 non-null object
vote_count              10866 non-null int64
vote_average            10866 non-null float64
release_year            10866 non-null int64
budget_adj              1[...]<jupyter_text>In this case we can see that some columns like 'homepage' and 'cast' they have missing values. So this means that I will have to impute this missing values or drop this rows.  Data CleaningFirst of all I will remove the columns that they are not useful for the analysis.<jupyter_code>#Get the list of the names of the columns
df.columns<jupyter_output><empty_output><jupyter_text>If I check the preview that I did before I can see that there are some columns that they are not usefull for this analysis, like for example 'id', 'idmb_id' and 'homepage'. So I will drop this columns before doing the analysis.<jupyter_code>#Drop the useless columns
df.drop(['id','imdb_id','homepage','overview'], axis = 1, inplace = True)

#Check again the name of the columns
df.head(2)<jupyter_output><empty_output><jupyter_text>Now we also can see the column 'keywords', in the first preview this column was hiden because the number of columns was too big.Another thing we can see from this dataset is that there are some columns that they contain multiple values splited by a vertical bar '|'. This means that later I will have to split this columns into multiple rows to do the analysis.Once I have removed the columns I will not use let's check again the number of NA for each column.<jupyter_code>#Check the number of NA and datatype of each column
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10866 entries, 0 to 10865
Data columns (total 17 columns):
popularity              10866 non-null float64
budget                  10866 non-null int64
revenue                 10866 non-null int64
original_title          10866 non-null object
cast                    10790 non-null object
director                10822 non-null object
tagline                 8042 non-null object
keywords                9373 non-null object
runtime                 10866 non-null int64
genres                  10843 non-null object
production_companies    9836 non-null object
release_date            10866 non-null object
vote_count              10866 non-null int64
vote_average            10866 non-null float64
release_year            10866 non-null int64
budget_adj              10866 non-null float64
revenue_adj             10866 non-null float64
dtypes: float64(4), int64(5), object(8)
memory usage: 1.4+ MB<jupyter_text>As I can see there are 3 columns that they have more than 1000 missing values, this are 'tagline', 'keywords' and 'production_companies'. Even that this columns can be really interestin to do some analysis, in this analysis they will not be needed to answer the questions that I want to answer. So insted of losing data from other columns, I will remove this 3 columns.<jupyter_code>#Drop the 3 columns with more NA
df.drop(['tagline', 'keywords', 'production_companies'], axis = 1, inplace = True)

#Check the header again
df.head(1)<jupyter_output><empty_output><jupyter_text>Finally I will remove the rows that they have NA so I will make sure that all the rows contains values for each column.<jupyter_code>#Check the number of columns before drop all NA
df.shape
#Drop all the rows with NA
df.dropna(inplace = True)

#Check again the number of columns after the drop
df.shape<jupyter_output><empty_output><jupyter_text>As we can see after the cleaning we have lost 134 rows and 7 columns.Finally I will check if there are some duplicated rows, so then I will remove this that are duplicated.<jupyter_code>#Check if there are duplicated rows
sum(df.duplicated())<jupyter_output><empty_output><jupyter_text>As we can see there is 1 duplicated row so I will remove this one before doing the EDA.<jupyter_code>#Drop duplicated rows
df.drop_duplicates(inplace = True)<jupyter_output><empty_output><jupyter_text>Before going to the EDA let's do a final check for all the columns.<jupyter_code>#Check the number of NA and datatype of each column
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 10731 entries, 0 to 10865
Data columns (total 14 columns):
popularity        10731 non-null float64
budget            10731 non-null int64
revenue           10731 non-null int64
original_title    10731 non-null object
cast              10731 non-null object
director          10731 non-null object
runtime           10731 non-null int64
genres            10731 non-null object
release_date      10731 non-null object
vote_count        10731 non-null int64
vote_average      10731 non-null float64
release_year      10731 non-null int64
budget_adj        10731 non-null float64
revenue_adj       10731 non-null float64
dtypes: float64(4), int64(5), object(5)
memory usage: 1.2+ MB<jupyter_text>Exploratory Data Analysis Which genres are most popular from year to year? To answer this question I will use the column 'popularity', 'release_year' and 'genres'. The tricky thing in this case is that the genres column contains multiple values splited by a vertical bar. This means I will have to split this column into multiples rows.<jupyter_code>#Select the columns I will use
popular_year = df[['popularity','release_year','genres']]

#Show the head of the table
popular_year.head(3)<jupyter_output><empty_output><jupyter_text>Before continuo I also would like to explore the variable 'release_year' to know how many years and films there are per year.<jupyter_code>#Count the number of films for each year
count_year = popular_year[['popularity', 'release_year']].groupby(['release_year'], as_index = False).count()

#Rename column 'popularity' to 'n'
count_year.rename(columns={'popularity': 'n'}, inplace = True)

#Show the header
count_year.head()<jupyter_output><empty_output><jupyter_text>Once I have a table with the number of fils for each year I will do a plot visualitzation to see better how many films there are for each year.<jupyter_code>#Create plot with the number of fils for each year
plt.plot(count_year.release_year, count_year.n)
plt.xlabel("Year")
plt.ylabel("Number of films")
plt.title("Number of films for each year")
plt.show()<jupyter_output><empty_output><jupyter_text>As I can see from the previous plot, the number of films has increased exponentially during the last years.Another nice thing to see it would be to see the film with the maximum popularity each year, just to see if there is some years that the popularity went really bad or even if there is a trend in the fils in where every x years there is a film with a really high popularity. To do this I will do it kind of similar than before.<jupyter_code>#Select for each year the maximum popularity
max_popularity_year = df[['popularity', 'release_year']].groupby(['release_year'], as_index = False).max()

#Show the head
max_popularity_year.head()
#Show the tail
max_popularity_year.tail()<jupyter_output><empty_output><jupyter_text>Now let's plot this data in a lineplot to see better what is the max popularity for each year.<jupyter_code>#Create plot with the max popularity for each year
plt.plot(max_popularity_year.release_year, max_popularity_year.popularity)
plt.xlabel("Year")
plt.ylabel("Popularity")
plt.title("Max popularity for each year")
plt.show()<jupyter_output><empty_output><jupyter_text>As I can see from the previous plot, the popularity of the movies has increased really a lot specially the last 2 years in where the max popularity it is above 24, meanwhile the popularity in the previous years was lower than 15.So I know the popularity and the amount of films per year let's go to answer the main question. To do that, first I need to split the column 'genres' into multiple rows (one row for each genre in each row).To do that I used the recommendation of this [link](https://stackoverflow.com/questions/39504079/take-column-of-string-data-in-pandas-dataframe-and-split-into-separate-columns).<jupyter_code>#Split the column 'genres' into multiple columns
popular_year_split = popular_year.set_index(['popularity','release_year']).genres.str.split('|', expand=True).reset_index()

#Show the header
popular_year_split.head()<jupyter_output><empty_output><jupyter_text>As I can see there is maximum 5 different type of genres for each film. Now I will melt all the dataframe and create a row for each genre and film. I know melt because I know quite well R and I use it with the library reshape2 to melt a dplyr dataframe.<jupyter_code>#Melt the dataframe
popular_year_split_melt = pd.melt(popular_year_split, id_vars = ['popularity','release_year'])

#Show the header
popular_year_split_melt.head()<jupyter_output><empty_output><jupyter_text>Now I will drop the column 'variable' because it's not useful for our analysis.<jupyter_code>#Drop column 'variable'
popular_year_split_melt.drop(['variable'], axis = 1, inplace = True)

#Show the header
popular_year_split_melt.head()<jupyter_output><empty_output><jupyter_text>Finally I will aggregate per year and genre by summing the popularity and finally I will take the max of each year.<jupyter_code>#Agregate by year and genre and sum the popularity
genre_year = popular_year_split_melt.groupby(['release_year','value']).sum().reset_index()

#Show head
genre_year.head()<jupyter_output><empty_output><jupyter_text>Now that I have the table with the total sum of popularity for each year and genre I will choose the genre with the highest popularity for each year. To do that, I found some help in this [link](https://stackoverflow.com/questions/41815079/pandas-merge-join-two-data-frames-on-multiple-columns).<jupyter_code>#Select the highest popularity of each year
list_highest_popularity_year = genre_year.groupby(['release_year']).agg({'popularity' : 'max'}).reset_index()

#Join the list of the highest popularity of each year with the list of 'genre_year' to get the name of the genre of each year
list_highest_popularity_year = pd.merge(genre_year, list_highest_popularity_year, how = 'inner', 
                                        left_on = ['release_year','popularity'], right_on = ['release_year','popularity'])

#Show the head
list_highest_popularity_year.head()<jupyter_output><empty_output><jupyter_text>Finally I got the table in where I can see for each year which genre was more popular for that year.Now I will pivot this table and create a column for each different genre and then plot in a bar plot for each year. To this I found some help in this [link](https://stackoverflow.com/questions/48958035/pandas-convert-some-rows-to-columns-in-python) in where it is explained how to do a pivot and also in this [link](https://pandas.pydata.org/pandas-docs/version/0.15.0/visualization.html) in where I can do a stacked bar plot.<jupyter_code>#Stacket bar plot in where I can see the genre most popular for each year and the popularity of each genre for each year
list_highest_popularity_year.rename(columns = {'value': 'genre'})\
.pivot(index='release_year', columns='genre', values='popularity').reset_index()\
.plot(kind='bar',x='release_year', stacked=True, figsize=(10,5));
plt.xlabel("Year");
plt.ylabel("Popularity");
plt.title("Genre most popular for each year and the total sum of popularity");
plt.show();<jupyter_output><empty_output><jupyter_text>In the previous plot we can see clearly that the genre that is more popular almost all the years is the 'Drama'.If we want to see this more clear, I will aggragate the data and count the total for each genre.<jupyter_code>list_highest_popularity_year[['value','popularity']].groupby(['value']).count().reset_index()\
.plot(kind='bar',x='value',y='popularity');
plt.xlabel("Genre");
plt.ylabel("Nº years");
plt.title("Nº of years that each genre is the most popular from 1960 until 2015");
plt.show();<jupyter_output><empty_output><jupyter_text>As I can see, the winner is the genre 'Drama' with 39 times and the next one is 'Comedy' with 9.  What kinds of properties are associated with movies that have high revenues?To asnwer this question it gets more tricky so first let's see the columns and see which of them can be interesting to answer this question.<jupyter_code>#Show the header
df.head(3)<jupyter_output><empty_output><jupyter_text>The first thing to take into account is that to answer this questions we can do it using numerical variables and categorical variables. For example, can be that some actors or directors they maybe have a high probavility to have a high revenue. Also can affect the genres and other factors.As we can see, there are two columns with the budget and the revenue. This is because during the last years the inflation makes the things more expensive so the columns that they end with `_adj` means that they are adjusted with the inflation. So to check the revenue I will use the adjusted columns.In this case, I will focus only with the numerical columns. This are the 'popularity', 'runtime', 'vote_count', 'release_year', 'budget_adj' and 'revenue_adj'.<jupyter_code>#Filter the original dataset with only the numerical columns
numeric_df = df[['popularity','runtime','vote_count','release_year','budget_adj','revenue_adj']].copy()

#Show the header
numeric_df.head()
#Show the information of this dataset
numeric_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 10731 entries, 0 to 10865
Data columns (total 6 columns):
popularity      10731 non-null float64
runtime         10731 non-null int64
vote_count      10731 non-null int64
release_year    10731 non-null int64
budget_adj      10731 non-null float64
revenue_adj     10731 non-null float64
dtypes: float64(3), int64(3)
memory usage: 586.9 KB<jupyter_text>As I can see the values in 'budget_adj' and 'revenue_adj' are really big so I will divide by 1 million and convert it to int.<jupyter_code>#Change the datatype of the column 'budget_adj' from float to int
numeric_df['budget_adj'] = (numeric_df['budget_adj'] / 1000000).astype(int)

#Change the datatype of the column 'revenue_adj' from float to int
numeric_df['revenue_adj'] = (numeric_df['revenue_adj'] / 1000000).astype(int)

numeric_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 10731 entries, 0 to 10865
Data columns (total 6 columns):
popularity      10731 non-null float64
runtime         10731 non-null int64
vote_count      10731 non-null int64
release_year    10731 non-null int64
budget_adj      10731 non-null int64
revenue_adj     10731 non-null int64
dtypes: float64(1), int64(5)
memory usage: 586.9 KB<jupyter_text>Now let's check again the data.<jupyter_code>#Show the header
numeric_df.head()<jupyter_output><empty_output><jupyter_text>The first thing I will do is create a histogram for each column to see the distribution of each feature.<jupyter_code>#Create a histogram for each feature
numeric_df.hist(figsize=(12,8), bins = 50);<jupyter_output><empty_output><jupyter_text>As I can see, in this case there is a lot of 0 in this data or at least low values. Using this data it will be dificult to see if there is a correlation between this features. So before looking for correlations, let's clean more this data.The first thing I will look is to check the number of films that they have 0 as a revenue.<jupyter_code>#Number of rows with the value of revenue 0
numeric_df.query('revenue_adj == 0.0').shape[0]<jupyter_output><empty_output><jupyter_text>As excepected, the number of rows in where the revenue is 0 is really big! So almost half of the films they don't have revenue, this in reality is really unlikely. So probably is missing this data and it was just imputed with a 0.Then I will remove this lines before checking the features that they are more correlated with the revenue. And also I will remove the lines with budget 0 because it also doesn't make sense to make a film without budget.<jupyter_code>#Drop the lines with revenue 0
numeric_df.drop(numeric_df.query('revenue_adj == 0.0').index, axis = 0, inplace = True)

#Drop the lines with budget 0
numeric_df.drop(numeric_df.query('budget_adj == 0.0').index, axis = 0, inplace = True)

#Show size
numeric_df.shape<jupyter_output><empty_output><jupyter_text>At the end I only have 3546 rows. Now let's plot again the histogram.<jupyter_code>#Create a histogram for each feature
numeric_df.hist(figsize=(12,8), bins = 50);<jupyter_output><empty_output><jupyter_text>Now I will do a scatter plot for each of them.To do that I will use the library seaborn in where I will be available to create a scatter plot for all the different features and also a bar plot. To do that I found the documentation in this [link](https://seaborn.pydata.org/generated/seaborn.pairplot.html).<jupyter_code>sns.set(style="ticks", color_codes=True);
sns.pairplot(numeric_df);<jupyter_output><empty_output><jupyter_text>As I can see from the previous plot, the feature `revenue_adj` doesn't have any column with a right strong correlation. Maybe the one is more correlated could be with the `vote_count`.To see better this scatterplot I will plot it individually using the library matplotlib.<jupyter_code>#Create a scatter plot
plt.scatter(x = numeric_df.revenue_adj, y = numeric_df.vote_count);
plt.xlabel("Revenue adjusted");
plt.ylabel("Vote count");
plt.title("Scatter plot of the revenue and the vote count");
plt.show();<jupyter_output><empty_output><jupyter_text>Another whey to see if there are correlations is using a correlation matrix. To do that I will use the function `corr()`. To do this I found the code in this [link](https://stackoverflow.com/questions/29432629/plot-correlation-matrix-using-pandas).<jupyter_code>#Correlation matrix of the dataframe 'numeric_df'
numeric_df.corr().style.background_gradient(cmap='coolwarm')<jupyter_output><empty_output><jupyter_text>ConclusionsAs a conclusion of the first question I can say that the genre most popular during the last years is the `Drama` followed by the `Comedy`. But the winner is clear with a big difference compared to the other gendres.For the second question, as I said before this question can be really general and big to answer because the revenue can depend in a lot of things and even I could use more external data to unser this questions but just to not make really big this project I choose only to use the numeric variables. With this variables I coudln't see a big correlations with the revenue. The highest one was 0.64 with the `vote_count` column follower by the `budget_adj` with 0.55. This doesn't mean that this are really key to determine the revenue of the film but it make sense because a film with a high budget and a lot of votes is high probability that it will have a big revenue. Submitting your Project<jupyter_code>from subprocess import call
call(['python', '-m', 'nbconvert', 'Investigate_a_Dataset.ipynb'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07222831006717804, 'fraction_numerical': 0.029968078517318596, 'mean_word_length': 4.634899328859061, 'pattern_counts': {'":': 0, '<': 141, '<?xml version=': 0, '>': 141, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '492791', 'license': 'MIT', 'n_tokens_mistral': 6532, 'n_tokens_neox': 5943, 'n_words': 2570, 'path': 'Investigate_a_Dataset.ipynb', 'repo_name': 'robertgv/Investigate_a_Dataset', 'size': '20989'}