<jupyter_start><jupyter_text>Project: Trends on the Film Industry Table of ContentsIntroductionData WranglingExploratory Data AnalysisConclusions  Introduction The selected dataset for this project is a curated one from Kaggle. This data set contains informationabout 10,000 movies collected from The Movie Database (TMDb),including user ratings and revenue. You can download it from here: https://www.kaggle.com/tmdb/tmdb-movie-metadata.Some of the questions that I want to explore with this analysis are:- Which genres are most popular from decade to decade?- Are revenue and budget associated with movies that have a high rating?<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import plotly
import plotly.express as px
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Structure and characteristics of the datasetIn this section of the report, I will explore the structure of the dataset and check for cleanliness, and then trim and clean the dataset for analysis.  General Properties<jupyter_code>#The dataset is loaded with Pandas
df = pd.read_csv("tmdb-movies.csv")
df.head(3)
# Type information for each column and non-null exploration
df.info()
#Let's check how many columns and attributes the data set has
df.shape<jupyter_output><empty_output><jupyter_text>Some of the insights it is possible to gather from the following table are:- There are some films with 0 budget and revenue. These values should be missing or not be available.- The films on this dataset are between 1960 and 2015. - The mean popularity of all the film data sets is around 0.64.- The mean vote average is 5.97. This is the mean score is around 6 points against 10.<jupyter_code># Statistical approach of the data set
df.describe()
# Let's analyze how many NaN values there are in every attribute
df.isna().sum()<jupyter_output><empty_output><jupyter_text>Analyzing duplicates In this step, I will be analyzing if there are duplicated values on the dataset.<jupyter_code># Let's see if there are duplicated values in the dataset
df.duplicated(keep = 'last').value_counts()
# Removing the duplicated row from the dataset
df.drop_duplicates(inplace = True)
# Checking again the shape of the dataset. One value has been removed.
df.shape<jupyter_output><empty_output><jupyter_text>Data Cleaning: Missing Values, Dropping Columns After discussing the structure of the data and any problems that need to be cleaned, let's perform those cleaning steps in this section.  Dropping Columns<jupyter_code># Let's drop the columns that are not going to be needed
df.drop(["imdb_id", "homepage", "keywords", "tagline"], axis = 1, inplace = True)<jupyter_output><empty_output><jupyter_text>Imputing missing values<jupyter_code># Let's analyze how many NaN values there are in every attribute after dropping columns
df.isna().sum()<jupyter_output><empty_output><jupyter_text>There are some categorical variables still that have missing values. I have decided no to use the mode because it is not representative in this case. For example, it is unlikely that the director that has appeared more times in the dataset was going to film another 44 films in the dataset (number of missing values in the attribute Director). I am going to impute the missing values in the five columns in the following way:- Cast will be filled with 'No cast available'.- Director will be filled with 'No director available'.- Overview will be filled with 'No overview available'.- Genres will be filled with 'No gender available'.- Production_company will be filled with 'No production company available'.<jupyter_code>df['cast'].replace(np.nan, 'No cast available', inplace = True)
df['director'].replace(np.nan, 'No director available', inplace = True)
df['overview'].replace(np.nan, 'No overview available', inplace = True)
df['genres'].replace(np.nan, 'No genres available', inplace = True)
df['production_companies'].replace(np.nan, 'No production company available', inplace = True)
# Let's analyze the NaN values again just to confirm there are no more missing values on the dataset
df.isna().sum()<jupyter_output><empty_output><jupyter_text>Cleaning genres, actors and production_companies columns  Genres column A film can have multiple genres. Films often have genres that overlap, such as adventure in a spy film, or crime in a science fiction film. But one genre is predominant. For this reason, only the first genre will be kept.<jupyter_code>df.head(2)
# Let's split using the symbol | and create a new dataframe with all the genres separatelly, keeping just the first column
df_genres = df['genres'].str.split("|", expand = True)
df_genres.drop([1,2,3,4], axis = 1, inplace = True)
df_genres.head(3)
# Let's concatenate both dataframes, change the name of column 0 and drop the column 'genres'
dft = pd.concat([df.reset_index(drop=True), df_genres.reset_index(drop=True)], axis=1)
dft.rename(columns={0: 'main_genre'}, inplace = True)
dft.drop('genres', axis = 1, inplace = True)
dft.head(3)<jupyter_output><empty_output><jupyter_text>Let's see how many different main genres we have now listed on the dataset:<jupyter_code>dft['main_genre'].value_counts()<jupyter_output><empty_output><jupyter_text>Cast & production companies columns Let's replace the symbol | that appears on both columns on this step.<jupyter_code>dft['cast'] = dft['cast'].str.replace('|', ", ")
dft['production_companies'] = dft['production_companies'].str.replace('|', ", ")
dft<jupyter_output><empty_output><jupyter_text>Exploratory Data AnalysisLet's move on to exploratory data analysis in this section. First, I will look carefully at the variables involved in each question, and then I will analyze the relationship between them.  What Genres Have Been More Popular over the Decades? First of all, let's see which genres are the most popular:<jupyter_code>fig = plt.subplots(figsize=(26,8))
sns.countplot(data = dft, x = "main_genre").set_title("Films by Genre")
plt.show()<jupyter_output><empty_output><jupyter_text>As we can see, five genders have more films: Drama, Comedy, Action, Honor, and Adventure. Let's see what happens over the years. As you can see in the following graphic, every year more and more films are released. This growth starts at the very beginning of 1980 and continues until 2015. Why is this happening? It would be necessary more information to get conclusions, but the technology makes it easier than every record your films with a low budget.<jupyter_code>fig = plt.subplots(figsize=(26,8))
sns.countplot(data = dft, x = "release_year").set_title("Films Released by Year")
plt.show()
#Let's create a dataframe grouped by release_year and main_genre
dft_g = dft.groupby(['release_year', 'main_genre']).count()
dft_g.reset_index(inplace = True)
dft_g
#Creating a graphic when it is possible to see the evolution of every genre over the years
fig = px.line(dft_g, x="release_year", y="id", color='main_genre', title = "Evolution of Genre by Year")
fig.show()<jupyter_output><empty_output><jupyter_text>On the last graph, we can see some trends about the genre over decades:    - Drama is the main genre that has been growing more every decade. It is the most used genre in films.- The number of films categorized as Comedy has been decreasing since 2010.- Thriller is a genre that has been growing faster since 2010.- Horror is another of the genres that have been rising since 1998.  Are revenue and budget associated with movies that have a high rating? With this question, I want to explore if there is a relationship between these two variables against rating. Does a high revenue or budget imply a high rating or vice-versa? Or is not there any correlation? First of all, let's see how these variables behave separately with a histogram. A filter will be applied to select only the rows with revenue different to 0 as it is possible to understand there is no data available for those films or they weren't released on cinemas.<jupyter_code>dft_nz = dft[(dft['revenue_adj'] != 0)]
# Let's see the distribution of revenue_adj
fig = plt.subplots(figsize=(10,6))
sns.histplot(data=dft_nz, x="revenue_adj").set_title("Distribution of revenue")
#Let's see the distribution of vote_average
sns.histplot(data=dft_nz, x="vote_average").set_title("Distribution of rating")<jupyter_output><empty_output><jupyter_text>We can see that most of the films have a rating of around 5.8 and 6.5. Also, most of the films have had revenue around 0.0 and 0.5 million dollars.  Now, let's create a scatter plot to see the behaviour between these two attributes:<jupyter_code>sns.jointplot(dft_nz["revenue_adj"],dft_nz["vote_average"])
plt.suptitle("Correlation between rating and revenue")<jupyter_output>C:\Users\xsof\anaconda3\lib\site-packages\seaborn\_decorators.py:36: FutureWarning:

Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.<jupyter_text>It is not possible to see any pattern on the scatter plot graphed. This means that there is no relationship between the variables. Let's confirm our hypotheses with a correlation graphic between both variables:<jupyter_code>sns.heatmap(dft_nz[["vote_average","revenue_adj"]].corr(), annot=True, cmap="Blues").set_title("Correlation between rating and revenue")<jupyter_output><empty_output><jupyter_text>As we can see, the correlation between these two variables is low as it is near to 0. It can be assumed that there is no relationship between attributes in this case.Does this relationship of no correlation happen with budget_adj? Let's see it:<jupyter_code>#Let's see the distribution of budget_adj
sns.histplot(data=dft_nz, x="budget_adj").set_title("Distribution of budget_adj")<jupyter_output><empty_output><jupyter_text>We can see that there are lots of films that have assigned 0 as budget. Also, we can see there is no patterns on the scatter plot:<jupyter_code>sns.jointplot(dft_nz["budget_adj"],dft_nz["vote_average"])
plt.suptitle("Correlation between rating and budget")<jupyter_output>C:\Users\xsof\anaconda3\lib\site-packages\seaborn\_decorators.py:36: FutureWarning:

Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.<jupyter_text>And, again, it is possible to confirm that there is no correlation between rating and budget, as we see in the graph below:<jupyter_code>sns.heatmap(dft_nz[["vote_average","budget_adj"]].corr(), annot=True, cmap="Blues").set_title("Correlation between rating and budget")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08140507387789239, 'fraction_numerical': 0.008549391320509246, 'mean_word_length': 5.872286079182631, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 71, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '122997', 'license': 'CC0-1.0', 'n_tokens_mistral': 3179, 'n_tokens_neox': 2997, 'n_words': 1323, 'path': 'EDA - Film Industry Trends.ipynb', 'repo_name': 'SofiaValan/eda-trends-film-industry', 'size': '10761'}