<jupyter_start><jupyter_text>> **Tip**: Welcome to the Investigate a Dataset project! You will find tips in quoted sections like this to help organize your approach to your investigation. Before submitting your project, it will be a good idea to go back through your report and remove these sections to make the presentation of your work as tidy as possible. First things first, you might want to double-click this Markdown cell and change the title so that it reflects your dataset and investigation. Project: Investigate a Dataset (Replace this with something more specific!) Table of ContentsIntroductionData WranglingExploratory Data AnalysisConclusions  Introduction> **Tip**: In this section of the report, provide a brief introduction to the dataset you've selected for analysis. At the end of this section, describe the questions that you plan on exploring over the course of the report. Try to build your report around the analysis of at least one dependent variable and three independent variables.>> If you haven't yet selected and downloaded your data, make sure you do that first before coming back here. If you're not sure what questions to ask right now, then make sure you familiarize yourself with the variables and the dataset context for ideas of what to explore. DataSet DescriptionThis data set ('tmdb-movies.csv') contains information about 10,000 movies collected from The Movie Database (TMDb), including user ratings and revenue.  QuestionsAs checked in the below cells, after studing the data, there are few key aspects which would like to explore in this project:* Which type of movie genre has the heighest and the lowest vote_average?* What is the tend of budget for making a movie over time, lets see this via 'budget_adj' field?* Which actors have the highest summed revenue for their movies? * Which directors have the highest summed revenue for their movies? * What is the Corelation between popularity of movie and its vote average?* What is the Corelation of vote_average with respect to runtime, vote_count, revenue_adj or budget_adj?<jupyter_code># Use this cell to set up import statements for all of the packages that you
#   plan to use.

# Remember to include a 'magic word' so that your visualizations are plotted
#   inline with the notebook. See this page for more:
#   http://ipython.readthedocs.io/en/stable/interactive/magics.html

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# visualizations are plotted inline with the notebook
% matplotlib inline<jupyter_output><empty_output><jupyter_text>Data Wrangling> **Tip**: In this section of the report, you will load in the data, check for cleanliness, and then trim and clean your dataset for analysis. Make sure that you document your steps carefully and justify your cleaning decisions. General Properties<jupyter_code># Load data into DataFrame
df = pd.read_csv('tmdb-movies.csv')
# Print out a few lines from the loaded DataFrame. 
df.head()
# Check the size of the data and number of Columns per record.
df.shape<jupyter_output><empty_output><jupyter_text>Above, we see that the total number of records(rows) are 10866.<jupyter_code>#Perform operations to inspect data types and look for instances of missing or possibly errant data.
df.dtypes<jupyter_output><empty_output><jupyter_text>Values like popularity, budget, revenue, runtime, vote_count, vote_average, budget_adj, revenue_adj are already in integer or float form, which in teran will help us in analysis. One value which I would like to change is 'release_date' which is currently string, lets change this to datetime object, which I'll do in data cleaning process below.  Lets also See the number of NaN entries in the dataframe:<jupyter_code># For below solution, I used this "https://stackoverflow.com/questions/39421433/efficient-way-to-find-null-values-in-a-dataframe"
# StackOverflow page for assistance.
def list_null_count(df):
    null_counts = df.isnull().sum()
    print ('Total Null entries with respect to each column:')
    return null_counts.sort_values(ascending=False)  
    
list_null_count(df)<jupyter_output>Total Null entries with respect to each column:<jupyter_text>By above data, we can see that there are some columns which have a significant count of missing values. But all of the columns which have int or fload data have values in them.In below cell lets count the number of unique values in each column:<jupyter_code># https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.nunique.html
df.nunique()<jupyter_output><empty_output><jupyter_text>In above cell's output we can see that there are lot of unique data, one thing we notice that there are total 56 unique year's data we have got.Additionally if you see, the total number of Records are 10866, but unique 'id' are 10865, which means there is one record which is repeated. We will remove that in our data cleaning process.<jupyter_code># Show description of Data:
df.describe()<jupyter_output><empty_output><jupyter_text>* Runtime varies from 0 to 900* vote_average varies from 10 to 9767* Overall all the movies are from year 1960 to 2015 > **Tip**: You should _not_ perform too many operations in each cell. Create cells freely to explore your data. One option that you can take with this project is to do a lot of explorations in an initial notebook. These don't have to be organized, but make sure you use enough comments to understand the purpose of each code cell. Then, after you're done with your analysis, create a duplicate notebook where you will trim the excess and organize your steps so that you have a flowing, cohesive report.> **Tip**: Make sure that you keep your reader informed on the steps that you are taking in your investigation. Follow every code cell, or every set of related code cells, with a markdown cell to describe to the reader what was found in the preceding cell(s). Try to make it so that the reader can then understand what they will be seeing in the following cell(s). Data Cleaning (Replace this with more specific notes!)To clean data we will perform:* Remove duplicate data* Change 'release_year' from string to datetime object* Replace NaN in genres, cast, director, and tagline with empty string.* Seperate multiple values separated by pipe (|) characters in python lists Remove Duplicate dataAs we have checked aboove, there was one row which was duplicate, so lets first remove that:<jupyter_code># After discussing the structure of the data and any problems that need to be
#   cleaned, perform those cleaning steps in the second part of this section.

#Step 1: show count of duplicate rows (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.duplicated.html):
print ("Total duplicate rows: {}".format(sum(df.duplicated())))<jupyter_output>Total duplicate rows: 1<jupyter_text>Now lets delete this duplicate row:<jupyter_code># Step 2: Remove duplicate: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html
df = df.drop_duplicates()
print ("Total duplicate rows after drop_duplicates: {}".format(sum(df.duplicated())))<jupyter_output>Total duplicate rows after drop_duplicates: 0<jupyter_text>Lets fix data type by conversion:<jupyter_code>df['release_date'] = pd.to_datetime(df['release_date'])

# Lets check the data types to confirm that the date type is changed
df.dtypes<jupyter_output><empty_output><jupyter_text>"release_date" column is changed from "Object" to "datetime64[ns]", Now, lets replace NaN to Empty strings:<jupyter_code># check Nan info: genres, tagline and cast
df.info()
# Lets now clear the NaNs in the columns we are interested in:
#https://stackoverflow.com/questions/26837998/pandas-replace-nan-with-blank-empty-string

def replace_NaN_by_empty_string(data, lable):
    data[lable] = data[lable].fillna('')
    return data
df = replace_NaN_by_empty_string(df, 'genres')
df = replace_NaN_by_empty_string(df, 'tagline')
df = replace_NaN_by_empty_string(df, 'cast')

# check Nan info
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 10865 entries, 0 to 10865
Data columns (total 21 columns):
id                      10865 non-null int64
imdb_id                 10855 non-null object
popularity              10865 non-null float64
budget                  10865 non-null int64
revenue                 10865 non-null int64
original_title          10865 non-null object
cast                    10865 non-null object
homepage                2936 non-null object
director                10821 non-null object
tagline                 10865 non-null object
keywords                9372 non-null object
overview                10861 non-null object
runtime                 10865 non-null int64
genres                  10865 non-null object
production_companies    9835 non-null object
release_date            10865 non-null datetime64[ns]
vote_count              10865 non-null int64
vote_average            10865 non-null float64
release_year            10865 non-null int64
budget_adj      [...]<jupyter_text>we can see that the Nans in the genres, tagline and cast are removed  Exploratory Data Analysis> **Tip**: Now that you've trimmed and cleaned your data, you're ready to move on to exploration. Compute statistics and create visualizations with the goal of addressing the research questions that you posed in the Introduction section. It is recommended that you be systematic with your approach. Look at one variable at a time, and then follow it up by looking at relationships between variables. Question: Which type of movie genre has the heighest and the lowest vote_average?<jupyter_code># Lets assume that the first genre in the list is the most releavent. Assuming that, lets create a new column 'releavent_genre',
# which will contain the first genre:
def split_and_take_first_item(x):
    return x.split('|')[0]

df['releavent_genre'] = df['genres'].apply(split_and_take_first_item)

#Lets cheack if the new column is added:
df.head()<jupyter_output><empty_output><jupyter_text>Based on the new column, now let's see which genre has the highest and lowest vote_average':<jupyter_code>releavent_genre_sorted_for_mean = df.groupby('releavent_genre')['vote_average'].mean().sort_values(ascending=False) 
releavent_genre_sorted_for_mean
# Lets plot a bargraph of this data:
releavent_genre_sorted_for_mean.plot(kind='bar')<jupyter_output><empty_output><jupyter_text>We can clearly see that * The Gerne with the highest vote_average is "Documentry" followed by "Music".* The Gerne with the lowest vote_average is "Horror" followed by "Thriller".  Research Question: What is the tend of budget for making a movie over time, lets see this via 'budget_adj' field?<jupyter_code># Lets first create a dataframe from given df which has columns where value of budget_adj is not equal to 0
df_budget_adj = df[df.budget_adj != 0]
df_budget_adj.shape
# Total Movies count
df_budget_adj.groupby('release_year')['budget_adj'].count().plot()
# Mean adjusted budget over the years
df_budget_adj.groupby('release_year')['budget_adj'].mean().plot()
# Sum of budget put into movies every year.
df_budget_adj.groupby('release_year')['budget_adj'].sum().plot()<jupyter_output><empty_output><jupyter_text>We can see in the above two graphs that adjusted budget mean value had a diffent trend at different time, but to understand this when we see the total number of moved made in that perticuler year, we see that over time the number of movies made each and every ear is increasing, which in retun effect the mean of the adjusted budget over the years. So finally we check the total sum put into making of movies over the year, as the data is I've thought, increasing rapidly over time. We can say that adjusted budget mean over the years has increased and decreased since lot and lot of movies are being made each and every coming year. Sinxce 2000 , the number of movies per year has significantly increased, to the mean has shifted and pulled the average adjusted mean down.  Research Question: Which actors have the highest summed revenue for their movies?<jupyter_code># Similar to what we have done in Gerne, we'll take the first actor/actress form the cast assuming as them
# the lead character in the movie, so lets create a new column whith the most releavent actor or cast

df['releavent_cast'] = df['cast'].apply(split_and_take_first_item)
df.head()
# Since we now have new column which gives the releavent/first cast, we will check the first/top 10 casts which are in biggest 
# revenue collection movies.
# We first group on "releavent_cast" and on revenue_adj column we sum them, sort them and get first 10 casts
releavent_cast_sorted_for_sum = df.groupby('releavent_cast')['revenue_adj'].sum().sort_values(ascending=False)[:10]
releavent_cast_sorted_for_sum
# Plot the values on bar graph
releavent_cast_sorted_for_sum.plot(kind='bar')<jupyter_output><empty_output><jupyter_text>We can see that the Tom Cruise and <NAME> are the twoi cast are which have got the highest revenue for their movies followed by harrison ford  Research Question: Which directors have the highest summed revenue for their movies?<jupyter_code>director_df_sum_revenue = df.groupby('director')['revenue_adj'].sum().sort_values(ascending=False)[:10]
director_df_sum_revenue
director_df_sum.plot(kind='bar')<jupyter_output><empty_output><jupyter_text>With the above graph, we can say that moves directed by <NAME> generate way more revenue than by other directors, followed by <NAME>.  Research Question: Which directors have the highest ratio of revenue generated to budget for their movies?<jupyter_code>director_df_sum_revenue_all = df.groupby('director')['revenue_adj'].sum()
director_df_sum_budget_all = df.groupby('director')['budget_adj'].sum()
# Calculate Efficiency: 
director_efficiency = director_df_sum_revenue_all / director_df_sum_budget_all

# Show top 10 records
print (director_efficiency[:10])
len(director_efficiency)
# 1. Replace Inifnity values to NaN
# 2. Drop NaN
director_efficiency = director_efficiency.replace([np.inf, -np.inf], np.nan).dropna()
director_efficiency = director_efficiency.sort_values(ascending=False)
director_efficiency[:10]
director_efficiency[:3].plot(kind='bar')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06561624649859944, 'fraction_numerical': 0.014775910364145658, 'mean_word_length': 5.007993268826252, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 73, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '322807', 'license': 'Apache-2.0', 'n_tokens_mistral': 4118, 'n_tokens_neox': 3854, 'n_words': 1771, 'path': 'investigate-dataset.ipynb', 'repo_name': 'Vinayrraj/Investigate-TMDbMovieData', 'size': '14300'}