<jupyter_start><jupyter_text>Udacity DSND Project 1: Blog Post  Table of Contents- Project overview- Business understanding- Data understanding- Data preparation- Data modelling and results evaluation- Conclusions  Project overview This is the Blog Post Project for the Data Scientist ND. We will be applying CRISP-DM approach to the TMDb movie dataset. This dataset contains information about 10,000 movies regarding the following: budget, revenue, user rating, vote, genre and release year. It covers the period from 1960 to 2015.The libraries used in this project include pandas, numpy, seaborn, matplotlib.  Business Understanding We will try to answer the following three questions:- Which genres are the most popular?- Which genres are associated with the highest revenues?- What movie properties have highest impact on revenue?<jupyter_code># import packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
# Load data and display few rows. 

movies = pd.read_csv('tmdb-movies.csv')
movies.head(10)<jupyter_output><empty_output><jupyter_text>Data understanding To understand the data, we need to walkthrough the dataset<jupyter_code>#Get modies information
movies.info()
movies.shape
movies.describe()<jupyter_output><empty_output><jupyter_text>**From the above results, we observe the following**- Columns count: 21- Rows count: 10866  Data preparation The following steps are applied to prepare and clean the data, inspired by what was presented in the classroom: - Delete columns that will not be used in the analysis.- Delete duplicate values.- Delete missing values and zero values in budget and revenue columns.- split the multivalue data in the genre column.<jupyter_code>#drop unnecessary columns
movies.drop(['homepage','tagline','overview','production_companies','release_date','vote_count','keywords','imdb_id','budget_adj','revenue_adj','cast'] , axis=1 , inplace=True)
movies.head()
movies.isnull().sum()<jupyter_output><empty_output><jupyter_text>From the above results, we observe that the genre column has 23 missing values that will be deleted.<jupyter_code>movies.dropna(inplace = True)
movies.isnull().sum()<jupyter_output><empty_output><jupyter_text>Next, we will use dropna to remove zeor values. And for that, we need to convert zero values to null values first. Zero values in datasets indicate values that have been intilised but no actual values were provided for them. For example, if we keep zroe budgets and revenue movies, this will negatively affect further analysis of the data, so these values must be dropped.<jupyter_code>movies['budget'] = movies['budget'].replace(0, np.NAN)
movies['revenue'] = movies['revenue'].replace(0, np.NAN)
movies.dropna(inplace = True)
#check
movies.isnull().sum()<jupyter_output><empty_output><jupyter_text>**Check for duplicate values**<jupyter_code>movies.duplicated().sum()<jupyter_output><empty_output><jupyter_text>**Drop it**<jupyter_code>movies.drop_duplicates(inplace= True)
#check
movies.duplicated().sum()
#Prepare the genre column
movies['genres'] = movies.genres.str.split("|",expand=True)
movies.head()
movies.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 3853 entries, 0 to 10848
Data columns (total 10 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   id              3853 non-null   int64  
 1   popularity      3853 non-null   float64
 2   budget          3853 non-null   float64
 3   revenue         3853 non-null   float64
 4   original_title  3853 non-null   object 
 5   director        3853 non-null   object 
 6   runtime         3853 non-null   int64  
 7   genres          3853 non-null   object 
 8   vote_average    3853 non-null   float64
 9   release_year    3853 non-null   int64  
dtypes: float64(4), int64(3), object(3)
memory usage: 331.1+ KB<jupyter_text>Data modelling and results evaluation  Question 1- Which genres are the most popular?<jupyter_code>#count number of movies per genre
number_movies= movies.groupby('genres').release_year.count()
number_movies
#Visualise the results
chart = number_movies.plot.barh(figsize = (6,6))
chart.set(title = 'Most Popular Genres')
chart.set_ylabel('Genre')
chart.set_xlabel('Number of movies');<jupyter_output><empty_output><jupyter_text>- The most popular genres are Drama, Comedy and Action whereas the least popular are TV movies, Western, War and History.  Question 2 - Which genres are associated with the highest revenues?<jupyter_code>top_revenue_genres = movies.groupby('genres').revenue.max()
top_revenue_genres
#Visualise the results
highest_revenue = top_revenue_genres.plot.pie(fontsize = 11,figsize=(10, 8))
highest_revenue.set(title = 'Movie revenues per genre');<jupyter_output><empty_output><jupyter_text>Highest revenues are associated with Action and Drama, which is not surprising as they are among the most popular as well.  Question 3 - What movie properties have highest impact on revenue?<jupyter_code># Find genres with above average revenue
revenue_ave = movies['revenue'].mean()
revenue_ave
higher_revenue = movies['revenue'] > revenue_ave
lower_revenue = movies['revenue'] <= revenue_ave
higher_revenue.value_counts()
#Create a function to make code related to plots more modular
def my_plot (title, x , y, legend):
    plt.title(title)
    plt.ylabel(y)
    plt.xlabel(x)
    if legend == True:
        plt.legend()


#Check if the release year has an impact on the average revenue
movies.groupby('release_year')['revenue'].mean().plot(kind='line');
my_plot('Revenue Average per Year','Revenue','Year',False);<jupyter_output><empty_output><jupyter_text>Although the curve is clumsy, we can see that movie revenues get higher as the years pass. This may indicate more interest in movies in recent years.<jupyter_code>#Check if the movie rating has an impact on revenues
movies.vote_average[lower_revenue].hist(alpha=0.7,color='maroon' , label='Lower revenue')
movies.vote_average[higher_revenue].hist(color ='lightblue' , label='Higher revenue')
my_plot('Revenue per Rating','Rating','Revenue',True);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09543163713217363, 'fraction_numerical': 0.02097219964233458, 'mean_word_length': 5.220424671385238, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '495357', 'license': 'FTL', 'n_tokens_mistral': 1944, 'n_tokens_neox': 1799, 'n_words': 626, 'path': 'TMDb_movie.ipynb', 'repo_name': 'tmsubait/Udacity-DSND', 'size': '6151'}