<jupyter_start><jupyter_text>**Get Started with Exploratory Data Analysis****Author:** <NAME>, Statistical Consultant with Information, Technology & Consulting**Purpose:** In this Jupyter notebook, I will be showing you how to perform Exploratory Data Analysis (EDA) using most popular data science python libraries, like `pandas`, `matplotlib`, `seaborn`.**Date:** 11-03-21  **Self-Introduction***  Statistical Consultant*  Take questions about Statistics & Statistical Software Packages (R, Python, SAS, Stata, SPSS)*  Email: <EMAIL>*  [Research guides](http://researchguides.dartmouth.edu/statapp_koujue)  **Dartmouth Machine Learning Club (DMLC)**The mission of DMLC is to build a platform for Dartmouth research community to share and learn about Machine Learning, Deep Learning, AI. Internal and external experts are invited to present at the DMLC meetings. Since the establishment of the interest group, more than 150 faculty, researchers, and students coming across the whole campus (Arts & Sciences, Tuck, Thayer, Geisel) have joined the club and many successful events have been organized and well-received – physical meetings, online video conference sessions, webinars, trainings, etc.Want to join DMLC? - Please click [here](https://teams.microsoft.com/l/team/19%3aa165ad0559744ded9f6e4f5fe988f82f%40thread.skype/conversations?groupId=c87b11fb-98d8-4af6-a8f5-e6d899473761&tenantId=995b0936-48d6-40e5-a31e-bf689ec9446f)!You may also want to join the Slack Workspace for DMLC: dart-machinelearning.slack.com.If you are interested in learning about Python, check out this Slack Workspace for Dartmouth Python Community: dart-pythoncommunity.slack.com  **Learning Objectives**Learn how to use some popular data science libraries, like `pandas`, `matplotlib`, `seaborn`, etc. to conduct EDA in Python.  **Exploratory Data Analysis (EDA)**EDA is an important and recommended step prior to the training of a (statistical/machine learning) model. EDA exams distribution of the data and the relationship between features via descriptive statistics and visualization techniques. EDA is typically done using numerical and graphical methods. <NAME> (Tukey, 1977) advocated the practice of exploratory data analysis (EDA) as a critical part of the scientific process.   **Types of Data**Prior to starting any calculation and visualization, it is very important to understand the types of data which are commonly seen in a statistical study. There are two main types of data: **numerical data (quantitative or continuous)** and **categorical data (qualitative or discrete)**. *   **Numerical data**: a type of data that represent measurements or counts and can be put into a logical order, such as height, weight, age, salary, number of children, etc*   **Categorical data**: a type of data that can take on one of a limited, and usually fixed number of possible values and may be used to group data,  such as gender, race, religion, marital status, educational level, etc.It's a good (common) practice to separate descriptive statistics and visualization for the two types of data.  **Load the Relevant Libraries** The following code loads (imports) the libraries(modules) which will be used for this session:<jupyter_code># NumPy is used for working with arrays. 
# It also has functions for working in domain of linear algebra, fourier transform, and matrices. 
import numpy as np
# pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool,
# built on top of the Python programming language. 
import pandas as pd
# Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python.
import matplotlib.pyplot as plt
# Seaborn is a Python data visualization library based on matplotlib. 
# It provides a high-level interface for drawing attractive and informative statistical graphics.
import seaborn as sns<jupyter_output><empty_output><jupyter_text>**Exploring the Auto-MPG dataset**We will use the [auto-mpg dataset](https://www.kaggle.com/uciml/autompg-dataset?select=auto-mpg.csv) for all the demos.  You can also find this dataset on the [UCI machine learning repository](https://archive.ics.uci.edu/ml/datasets/auto+mpg).  "The data concerns city-cycle fuel consumption in miles per gallon, to be predicted in terms of 3 multivalued discrete and 5 continuous attributes." (Quinlan, 1993)Attribute Information of the original dataset:1. mpg: continuous2. cylinders: multi-valued discrete3. displacement: continuous4. horsepower: continuous5. weight: continuous6. acceleration: continuous7. model year: multi-valued discrete8. origin: multi-valued discrete9. car name: string (unique for each instance)[Download](https://drive.google.com/file/d/1IAxWWTA8juZ_dCjR6fnawh34Tv5JfDoj/view?usp=sharing) and save a copy of the dataset to your local computer. The following code loads the dataset into a data frame:<jupyter_code># Load the auto-mpg dataset into a dataframe named 'df' and retrieve the column names and define the missing values
df = pd.read_csv("auto-mpg.csv", header='infer', na_values=['NA','?'])
# Print out the original column names
print(f'The original column names: \n {df.columns.values} \n')
# Rename two columns: "model year" -> "year"; "car name" -> "name"
df.rename(columns={"model year": "year", "car name": "name"}, inplace=True)
print(f'Double check after renaming two columns: \n {df.columns.values} \n')
# Print out the first 5 rows in the dataset
print(df[0:5])
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 398 entries, 0 to 397
Data columns (total 9 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   mpg           398 non-null    float64
 1   cylinders     398 non-null    int64  
 2   displacement  398 non-null    float64
 3   horsepower    392 non-null    float64
 4   weight        398 non-null    int64  
 5   acceleration  398 non-null    float64
 6   year          398 non-null    int64  
 7   origin        398 non-null    int64  
 8   name          398 non-null    object 
dtypes: float64(4), int64(4), object(1)
memory usage: 28.1+ KB<jupyter_text>Pre-EDA: Data Wrangling  Missing ValuesMissing values exists in most of real-world data.  Ideally, every row of data will have values for all columns.  However, this is rarely the case.  A common practice is to replace missing values with the mean or the median value for a numeric variable, while the mode (the most frequent value) for a categorical variable.  Most of the values are present in the auto-mpg database.  However, there are missing values in one column. Let's find out which column has missing values. The following code replaces any missing (NA or '?') values in that column with the median:<jupyter_code># Display all rows 
pd.set_option('display.max_rows', 0)
# Number of missing (null) values in each column
null_counts = df.isnull().sum()
print("Number of null values in each column:\n{}".format(null_counts))<jupyter_output>Number of null values in each column:
mpg             0
cylinders       0
displacement    0
horsepower      6
weight          0
acceleration    0
year            0
origin          0
name            0
dtype: int64<jupyter_text>From the above, we can see that only `horsepower` has missing values.<jupyter_code># Confirm the missing in `horsepower`
print(f"horsepower has missing ('na')? {pd.isnull(df['horsepower']).values.any()} \n")

# Fill missing values with the median of the column    
print("Filling missing values with median of the column \n")
med = df['horsepower'].median()
df['horsepower'] = df['horsepower'].fillna(med)
# Confirm all the missing replacement                 
print(f"horsepower has missing ('na')? {pd.isnull(df['horsepower']).values.any()} \n")
# You can also use the follow code the drop all missing values
# df = df.dropna()<jupyter_output>horsepower has missing ('na')? True 

Filling missing values with median of the column 

horsepower has missing ('na')? False<jupyter_text>OutliersOutliers are values that appear far away from majority of the data.  Sometimes outliers are simply human errors or systematic errors. However, sometimes, outliers may reflect reality and implicit some unknown phenomena. Usually,  outliers are the values that are more than two standard deviations from the mean.<jupyter_code># Remove all rows where the specified column is +/- sd standard deviations
def outliers_remover(df, col_name, sd):
    drop_rows = df.index[(np.abs(df[col_name] - df[col_name].mean())
                          >= (sd * df[col_name].std()))]
    df.drop(drop_rows, axis=0, inplace=True)<jupyter_output><empty_output><jupyter_text>The code below will drop every row from the auto-mpg dataset where the `mpg` column is more than two standard deviations above or below the mean.<jupyter_code># Drop outliers in mpg
print(f'Length before mpg outliers dropped: {len(df)}')
outliers_remover(df,'mpg',2)
print(f'Length after mpg outliers dropped: {len(df)}')

pd.set_option('display.max_columns', 0)
pd.set_option('display.max_rows', 5)
display(df)<jupyter_output>Length before mpg outliers dropped: 398
Length after mpg outliers dropped: 388<jupyter_text>Sorting a Data SetSorting dataset allows to order the rows in either ascending or descending order for one or more columns. The following code sorts the dataset by name and displays the first car.<jupyter_code>df = df.sort_values(by='name', ascending=True)
print(f"The first car is: {df['name'].iloc[0]}")
      
display(df)<jupyter_output>The first car is: amc ambassador brougham<jupyter_text>Transformation (Rescaling)* **Normalization** refers to the rescaling of variables (columns in a tabular data) to a range of [0,1]* **Standardization** refers to the rescaling of variables with zero mean and unit variance.The following code is used to normalize or standardize the column `mpg`.<jupyter_code>print("min is {:.2f}, max is {:.2f}, mean is {:.2f}, std is {:.2f}".format(df.mpg.min(), df.mpg.max(), df.mpg.mean(), df.mpg.std()))
mpg_orig = df.mpg
mpg_stdd = (df.mpg - df.mpg.mean())/df.mpg.std()
mpg_norm = (df.mpg - df.mpg.min())/ (df.mpg.max() - df.mpg.min())
compare_mpg = pd.DataFrame({'Original': mpg_orig, 'Standardized': mpg_stdd, 'Normalized': mpg_norm})
pd.set_option('display.precision',2)
display(compare_mpg)<jupyter_output>min is 9.00, max is 39.10, mean is 23.02, std is 7.26<jupyter_text>Concatenating Rows and ColumnsPython can concatenate rows and columns together to form new data frames.  The code below creates a new data frame from the **mpg** and **horsepower** columns from the auto-mpg dataset.  The program does this by concatenating two columns together.<jupyter_code># Create a new dataframe from name and horsepower
col_horsepower = df['horsepower']
col_mpg = df['mpg']
result = pd.concat([col_mpg, col_horsepower], axis=1)

display(result)<jupyter_output><empty_output><jupyter_text>The **concat** function can also concatenate rows together.  This code concatenates the first three rows and the last three rows of the auto-mpg dataset.<jupyter_code># Create a new dataframe from first 3 rows and last 3 rows
result = pd.concat([df[0:3],df[-3:]], axis=0)

pd.set_option('display.max_rows', 0)
display(result)<jupyter_output><empty_output><jupyter_text>Converting a Dataframe to a MatrixYou can use the **values** property of a data frame to convert the data to a matrix.<jupyter_code>df.values<jupyter_output><empty_output><jupyter_text>You might wish only to convert some of the columns, to leave out the `name` column, use the following code.<jupyter_code>df[['mpg', 'cylinders', 'displacement', 'horsepower', 'weight',
       'acceleration', 'year', 'origin']].values<jupyter_output><empty_output><jupyter_text>Grouping a Data SetGrouping is a typical operation on data sets.  Structured Query Language (SQL) calls this operation a "GROUP BY."  Programmers use grouping to summarize data.   For example, the following code will group cylinders by the average (mean).  This code will provide the grouping.  In addition to **mean**, you can use other aggregating functions, such as **sum** or **count**.<jupyter_code>cyl_g = df.groupby('cylinders')['mpg'].mean()
cyl_g<jupyter_output><empty_output><jupyter_text>In the below example we group the data by cylinders and then get the result for a specific cylinder.<jupyter_code>print("The grouped data are as follows:\n\n", df.groupby('cylinders').get_group(4))<jupyter_output>The grouped data are as follows:

       mpg  cylinders  displacement  horsepower  weight  acceleration  year  origin                      name
315  24.3          4         151.0        90.0    3003          20.1    80       1               amc concord
374  23.0          4         151.0        93.5    3035          20.5    82       1            amc concord dl
296  27.4          4         121.0        80.0    2670          15.0    79       1             amc spirit dl
21   24.0          4         107.0        90.0    2430          14.5    70       2               audi 100 ls
177  23.0          4         115.0        95.0    2694          15.0    75       2                audi 100ls
119  20.0          4         114.0        91.0    2582          14.0    73       2                audi 100ls
317  34.3          4          97.0        78.0    2188          15.8    80       2                 audi 4000
141  29.0          4          98.0        83.0    2219          16.5    74       2           [...]<jupyter_text>Saving a Dataframe to CSVYou can use the following code to save a copy of the data frame as a .csv file .<jupyter_code>import os
path = "."
filename_write = os.path.join(path, "auto-mpg-processed.csv")
df.to_csv(filename_write, index=False) 
print("Done")<jupyter_output>Done<jupyter_text>You can check the saved .csv file using the following code.<jupyter_code>processed = pd.read_csv("auto-mpg-processed.csv")
display(processed)<jupyter_output><empty_output><jupyter_text>EDA  **Descriptive Statistics for Numeric Columns**<jupyter_code>cols_num = ['mpg', 'displacement', 'horsepower', 'weight', 'acceleration']
round(df[cols_num].describe(), 2)
# Return Pearson product-moment correlation coefficients.
cm = np.corrcoef(df[cols_num].values.T)
cm.round(2)
# Alternatively, you can use the function `corr`
cm = round(df[cols_num].corr(), 2)
display(cm)<jupyter_output><empty_output><jupyter_text>**Visualizing the Numeric Columns**<jupyter_code># Choose the style of the plot. To check available styles using plt.style.available
plt.style.use('fivethirtyeight')
# %matplotlib inline sets the backend of matplotlib to the 'inline' backend: 
# With this backend, the output of plotting commands is displayed inline within frontends like the Jupyter notebook, 
# directly below the code cell that produced it. The resulting plots will then also be stored in the notebook document.
%matplotlib inline
# Plot pairwise relationships in a dataset.
sns.pairplot(df[cols_num], height=2.5)
# Automatically adjust plot parameters to give specified padding.
plt.tight_layout()
# Save the current figure.
plt.savefig('pairplot.png', dpi=300)
# Display all figures
plt.show()
# Set the font size relative to the figure
sns.set(font_scale=1.0)
# Plot rectangular data as a color-encoded matrix.
hm = sns.heatmap(cm,
                 cbar=True,
                 annot=True,
                 square=True,
                 fmt='.2f',
                 annot_kws={'size': 10},
                 yticklabels=cols_num,
                 xticklabels=cols_num)

plt.tight_layout()
plt.savefig('heatmap.png', dpi=300)
plt.show()
# Draw a box plot to show distributions with respect to categories.
sns.boxplot(x='cylinders', y='horsepower', data=df)
plt.savefig("boxplot.png", dpi = 300)<jupyter_output><empty_output><jupyter_text>**Descriptive Statistics for Categorical (Discrete) Columns**<jupyter_code># Misleading stats without defining categorical variables:
round(df.describe(), 2)<jupyter_output><empty_output><jupyter_text>Obviously, `cylinders`, `year`, `origin` are not supposed to be "numeric". So, they need to be **defined** as **categorical**.<jupyter_code># Define some columns (discrete) as "categorical"
cols_cat = ['cylinders', 'year', 'origin']
df[cols_cat]=df[cols_cat].astype('category')
df[cols_cat].describe()
# Display the frequency distribution for all the categorical variables
display(df.cylinders.value_counts(sort=True))
display(df.year.value_counts(sort=True))
display(df.origin.value_counts(sort=True))<jupyter_output><empty_output><jupyter_text>**Visualizing the Categorical Columns**<jupyter_code>sns.set(style='darkgrid')
plt.figure(figsize=(10,5))
# Barchart for "origin"
ax = sns.countplot(x='origin', data=df)
plt.savefig("barchart.png", dpi=300)
cylinder_counts = df['cylinders'].value_counts()
df2 = pd.DataFrame({'cylinder_type': cylinder_counts}, 
                     index = df.cylinders.unique()
                   )
# Pie chart for "cylinder type"
df2.plot.pie(y='cylinder_type', figsize=(10,10), 
                  explode = (0, 0, 0, 0.2, 0.2), autopct='%.2f%%')
plt.savefig("pie.png", dpi=300)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09778037383177571, 'fraction_numerical': 0.027570093457943926, 'mean_word_length': 4.066883693400414, 'pattern_counts': {'":': 2, '<': 81, '<?xml version=': 0, '>': 84, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '426756', 'license': 'MIT', 'n_tokens_mistral': 5493, 'n_tokens_neox': 5035, 'n_words': 1882, 'path': 'eda_tutorial/Get_Started_With_Exploratory_Data_Analysis_11032021.ipynb', 'repo_name': 'jmdelvecchio/WISP_arctic_hydro', 'size': '17147'}