<jupyter_start><jupyter_text>Python Exploratory Data Analysis Template  Introduction **What is Exploratory Data Analysis ?**Exploratory Data Analysis or (EDA) is understanding the data sets by summarizing their main characteristics often plotting them visually. This step is very important especially when we arrive at modeling the data in order to apply Machine learning. Plotting in EDA consists of Histograms, Box plot, Scatter plot and many more. It often takes much time to explore the data. Through the process of EDA, we can ask to define the problem statement or definition on our data set which is very important. **How to perform Exploratory Data Analysis**There are some common methods and plots that would be used in the EDA process.  1.   Data Exploration2.   Data Cleaning3.   Model Building4.   Presenting Results![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAm8AAADBCAYAAACZv1c/AAAgAElEQVR4nO1d4bGrOg/8SjjFnApSQRpIAVRAAWkgDaQAmkgbKSMl8P14s7kbHck2BGITdmc0774cMLZYy7JsrP+NgiAIgiAIwmbwv9oVEARBEARBEMoh500QBEEQBGFDkPMmCIIgCIKwIch5EwRBEARB2BC+ynl7PB7j4/H4yDWCIAiCIAg18DXO2/V6HQ+Hw1Pu9/ufa06nU/aavu+ffz+dTp+ouiAIgiAIQjG+xnm73+8vjtn1ev1zDf/9cDiMt9stec3lcvlE1QVBEARBEIrxNc7bOI7j8Xh8Ol5937/8zTp3nnNmr/GcO0EQBEEQhJr4KuftfD4/Ha/j8fjyt2EY/jhvXdclrwEej8d4Pp+zUhqpy11/uVye11jcbrex67qno3o8Hse+790l4NvtVlTvEif1fr+H91+<KEY>JQWzcS8ZFRWzeS+lz8n/urIAiCIAiC0CTkvAmCIAiCIGwIct4EQRAEQRA2BDlvgiAIgiAIG4KcN<KEY>5CYII=) **What data is being explored ?**> *use double space to begin new line*Dataset Name: World Happiness Report  Dataset Description: *The World Happiness Report is a landmark survey of the state of global happiness. The first report was published in 2012, the second in 2013, the third in 2015, and the fourth in the 2016 Update. The World Happiness 2017, which ranks 155 countries by their happiness levels, was released at the United Nations at an event celebrating International Day of Happiness on March 20th. The report continues to gain global recognition as governments, organizations and civil society increasingly use happiness indicators to inform their policy-making decisions. Leading experts across fields – economics, psychology, survey analysis, national statistics, health, public policy and more – describe how measurements of well-being can be used effectively to assess the progress of nations. The reports review the state of happiness in the world today and show how the new science of happiness explains personal and national variations in happiness.*    Dataset Source: Kaggle  Dataset Link: https://www.kaggle.com/unsdsn/world-happiness **What data questions are hoping to be answered after analysis?** 1.   Question12.   Question23.   Question34.   Question4 ---  1. Importing libraries Below are the libraries that are used in order to perform EDA (Exploratory Data Analysis). *Pathlib* - Object-oriented filesystem paths  *Sys* - System-specific parameters and functions  *OS* - Operating system dependent functionality  *Pandas* - Data analysis and manipulation tool  *Numpy* - Multi-dimensional arrays and matrices, and high-level mathematical functions  *Seaborn* - Statistical data visualizations  *Matplotlib* - Plotting visualizations<jupyter_code># Infrastructure
from pathlib import Path
import sys
import os

# Science and Data
import pandas as pd  
import numpy as np

# Plotting Tools
import seaborn as sns
import matplotlib.pyplot as plt

print('Import Libraries Complete')<jupyter_output>Import Libraries Complete<jupyter_text>Pandas and Plotting Setup Options<jupyter_code># Pandas
pd.set_option('display.max_rows', 100)

# Matplotlib
%matplotlib inline
plt.rcParams['figure.figsize'] = (20, 10)
plt.rcParams['axes.spines.left'] = False
plt.rcParams['axes.spines.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.bottom'] = False
plt.rcParams['xtick.bottom'] = False
plt.rcParams['xtick.labelbottom'] = True
plt.rcParams['ytick.labelleft'] = True
plt.rcParams.update({'font.size': 18})

print('Library Setup Options Complete')<jupyter_output>Setup Options Complete<jupyter_text>Path Setup Options<jupyter_code>'''
PROJECT_ROOT = !git rev-parse --show-toplevel
PROJECT_ROOT = Path(PROJECT_ROOT[0])

print('Path Setup Complete)
'''<jupyter_output><empty_output><jupyter_text>Machine Learning using Scikit-LearnPython Data Science Handbook  Introducing SciKit-Learn for Google Colab  [Link](https://colab.research.google.com/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/05.02-Introducing-Scikit-Learn.ipynbscrollTo=cXpxypZgp8xH)<jupyter_code>'''
import sklearn as sk

print('Machine Learning Library Complete')
'''<jupyter_output><empty_output><jupyter_text>---  2. Downloading files Loading the data into the pandas data frame is certainly one of the most important steps in EDA, as we can see that the value from the data set is comma-separated. So all we have to do is to just read the CSV into a data frame and pandas data frame does the job for us. External Data Sources   Local files: [Link](https://colab.research.google.com/notebooks/io.ipynb)  Google Drive: [Link](https://colab.research.google.com/notebooks/io.ipynbscrollTo=c2W5A2px3doP)  Kaggle: [Link](https://colab.research.google.com/github/corrieann/kaggle/blob/master/kaggle_api_in_colab.ipynbscrollTo=DsyV01gDaxls)  Github: [Link](https://colab.research.google.com/github/googlecolab/colabtools/blob/master/notebooks/colab-github-demo.ipynb)    *Github requires use or raw.githubusercontent.com url*  2.1. Local files<jupyter_code>'''
df = pd.read_csv(' ')
print('Import Data Complete')
'''<jupyter_output><empty_output><jupyter_text>2.2. URL files<jupyter_code>'''
url = (' ')
df = pd.read_csv(url)
print('Import Data Complete')
'''<jupyter_output><empty_output><jupyter_text>2.3. Kaggle files  2.3.1. Installing Kaggle API in Colab<jupyter_code>'''
!pip install kaggle
'''<jupyter_output><empty_output><jupyter_text>2.3.2. Authenticating with Kaggle using kaggle.json<jupyter_code>'''
from google.colab import files

uploaded = files.upload()

for fn in uploaded.keys():
  print('User uploaded file "{name}" with length {length} bytes'.format(
      name=fn, length=len(uploaded[fn])))
  
# Then move kaggle.json into the folder where the API expects to find it.
!mkdir -p ~/.kaggle/ && mv kaggle.json ~/.kaggle/ && chmod 600 ~/.kaggle/kaggle.json
'''<jupyter_output><empty_output><jupyter_text>2.3.3. Listing competitions<jupyter_code>'''
!kaggle competitions list
'''<jupyter_output><empty_output><jupyter_text>2.3.4. Downloading a dataset<jupyter_code>'''
!kaggle datasets download "center-for-policing-equity/data-science-for-good"
'''
'''
!unzip data-science-for-good.zip
'''
'''
basepath = "Dept_23-00089/23-00089_ACS_data/23-00089_ACS_race-sex-age/"
df = pd.read_csv(basepath + "ACS_15_5YR_DP05_with_ann.csv")
print('Import Data Complete')
'''<jupyter_output><empty_output><jupyter_text>2.4. Github files<jupyter_code>url = ('https://raw.githubusercontent.com/bentegviz/Explonatory_Data_Analysis/main/files/Happiness.csv')
df = pd.read_csv(url)

print('Import Data Complete')<jupyter_output>Import Data Complete<jupyter_text>3. Data Exploration  3.1. Exploring the data frame first and last 5 samples<jupyter_code># To display the top 5 rows 
df.head(5)               
# To display the botton 5 rows
df.tail(5)<jupyter_output><empty_output><jupyter_text>---  3.2. Checking the types of data Check for the datatypes to ensure proper usage of *string* (object), *int* (int64), and *float* (float64).<jupyter_code># To display the data types
df.dtypes<jupyter_output><empty_output><jupyter_text>3.2.1. Ensure dates are type *date*, numerical data is type *int* or *float*  3.3. Exploring data frame info<jupyter_code># To display the info
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 158 entries, 0 to 157
Data columns (total 12 columns):
 #   Column                         Non-Null Count  Dtype  
---  ------                         --------------  -----  
 0   Country                        158 non-null    object 
 1   Region                         158 non-null    object 
 2   Happiness Rank                 158 non-null    int64  
 3   Happiness Score                158 non-null    float64
 4   Standard Error                 158 non-null    float64
 5   Economy (GDP per Capita)       158 non-null    float64
 6   Family                         158 non-null    float64
 7   Health (Life Expectancy)       158 non-null    float64
 8   Freedom                        158 non-null    float64
 9   Trust (Government Corruption)  158 non-null    float64
 10  Generosity                     158 non-null    float64
 11  Dystopia Residual              158 non-null    float64
dtypes: float64(9), int64(1), object(2)
memory usage: 1[...]<jupyter_text>3.4. Exploring data frame description  Basic statistical details: count, mean, std, min, max, quartiles<jupyter_code># To display the statistical description
df.describe()<jupyter_output><empty_output><jupyter_text>3.4.1. Ensure the *min*, *max*, and *mean* values make sense for the dataset ---  3.5. Exploring data frame column names<jupyter_code># To display the column names
df.columns<jupyter_output><empty_output><jupyter_text>4. Data Cleaning  4.1. Dropping columns Check columns that can be ommitted from the dataset<jupyter_code>'''
df = df.drop(['Engine Fuel Type', 'Market Category', 'Vehicle Style', 'Popularity', 'Number of Doors', 'Vehicle Size'], axis=1)
df.head(5)
'''<jupyter_output><empty_output><jupyter_text>---  4.2. Renaming columns Check the readability of the remaining column names, improving the readability of the data set.<jupyter_code>'''
# To rename the columns
df = df.rename(columns={"Engine HP": "HP", "Engine Cylinders": "Cylinders", "Transmission Type": "Transmission", "Driven_Wheels": "Drive Mode","highway MPG": "MPG-H", "city mpg": "MPG-C", "MSRP": "Price" })
# To check the new column names
df.head(5)
'''<jupyter_output><empty_output><jupyter_text>---  4.3. Dropping duplicate rows Check the instance of duplicated data within the dataset.  Generally, the larger the dataset, the higher probability of having duplicates.<jupyter_code># To display dataset (rows, columns)
df.shape
# To display the duplicate rows as a new data frame
duplicate_rows_df = df[df.duplicated()]
print("Number of duplicate rows: ", duplicate_rows_df.shape)<jupyter_output>Number of duplicate rows:  (0, 12)<jupyter_text>Remove the duplicate rows if they exist and can be removed.<jupyter_code># To display the number of rows
df.count()<jupyter_output><empty_output><jupyter_text>The display above indicates ______ rows and removing _____ rows of duplicate data.<jupyter_code># To drop the duplicated data
df = df.drop_duplicates()
df.head(5)
# To count the number of rows after dropping duplicates 
df.count()<jupyter_output><empty_output><jupyter_text>---  4.4. Dropping missing or null values. This is mostly similar to the previous step but in here all the missing values are detected and are dropped later. Now, this is not a good approach to do so, because many people just replace the missing values with the mean or the average of that column, but in this case, I just dropped that missing values. This is because there is nearly 100 missing value compared to 10, 000 values this is a small number and this is negligible so I just dropped those values.<jupyter_code># To display null values
print(df.isnull().sum())
# To drop the null values
df = df.dropna()
df.count()<jupyter_output><empty_output><jupyter_text>Now we have removed all the rows which contain the Null or N/A values (Cylinders and Horsepower (HP)).<jupyter_code># To display the null values after dropping
print(df.isnull().sum())<jupyter_output>Country                          0
Region                           0
Happiness Rank                   0
Happiness Score                  0
Standard Error                   0
Economy (GDP per Capita)         0
Family                           0
Health (Life Expectancy)         0
Freedom                          0
Trust (Government Corruption)    0
Generosity                       0
Dystopia Residual                0
dtype: int64<jupyter_text>---  5. Model Building **Detecting Outliers**  5.1. Box plots Use box plots to detect data value outliers.<jupyter_code>'''
# To display box plots for all columns
for column in df.columns:
  plt.figure()
  sns.boxplot(x=df[column])
'''
# To display box plot for specific column
sns.boxplot(x=df['Happiness Score'])
'''
# To display box plot for specific column
sns.boxplot(x=df['column_name'])
'''<jupyter_output><empty_output><jupyter_text>Outliers are a point or set of points that are different from other points, sometimes they can *be* very high or very low.  Outliers are one of the primary reasons for resulting in a less accurate model we are going to remove them using the IQR score technique.  Outliers can be seen with visualizations using a Box Plot.   IQR score technique from [Towards Data Science](https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba).<jupyter_code># To display Quantile for IQR score
q1 = df.quantile(0.25)
q3 = df.quantile(0.75)
iqr_score = q3 - q1
print(iqr_score)
df = df[~((df < (q1 - 1.5 * iqr_score)) |(df > (q3 + 1.5 * iqr_score))).any(axis=1)]
df.shape<jupyter_output><empty_output><jupyter_text>---  5.2. HistogramsUse histogram display the frequency of occurrence of variables in an interval.<jupyter_code>'''
# To display histograms for all columns
df.columns.value_counts().nlargest(40).plot(kind='bar', figsize=(10,5))
'''
'''
# To display histogram for specific column
df.Economy.value_counts().nlargest(40).plot(kind='bar', figsize=(10,5))
plt.title("Number of cars by make")
plt.ylabel('Number of cars')
plt.xlabel('Make');
'''<jupyter_output><empty_output><jupyter_text>5.3. Heat MapsUse heat maps to find the dependent variables.<jupyter_code># To calculate the correlation coefficient matrix
df_corr = df.corr()
# To plot the Heat Map
sns.heatmap(df_corr, annot=True)
df_corr<jupyter_output><empty_output><jupyter_text>5.4. Scatter plotsUse scatter plots to find the correlation between two variables.<jupyter_code># To display scatter plot
fig, ax = plt.subplots(figsize=(10,6))
ax.scatter(df['column_name_a'], df['column_name_b'])
ax.set_xlabel('column_name_a')
ax.set_ylabel('column_name_b')
plt.show()<jupyter_output><empty_output><jupyter_text>5.5. Regression plots Use regression plots to analyze two variables.  *regplot()* or *lmplot()*<jupyter_code>'''
# To display regression plot
sns.lmplot(x='column_name_a', y='column_name_b', data=df)
'''
'''
# To display regression plot with hue
sns.lmplot(x='column_name_a', y='column_name_b', cue='column_hue' data=df)
'''<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09970813819317179, 'fraction_numerical': 0.026742686486119594, 'mean_word_length': 4.621518504387638, 'pattern_counts': {'":': 7, '<': 115, '<?xml version=': 0, '>': 116, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '576811', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5062, 'n_tokens_neox': 4738, 'n_words': 1464, 'path': 'notebooks/movie_dataset_pythonEDA.ipynb', 'repo_name': 'bentegviz/data_analyst_portfolio_project', 'size': '34943'}