<jupyter_start><jupyter_text>Statistical Foundations Module 2 – Exploratory Data AnalysisDemo 1: Detecting and Removing OutliersIn this demo, you will be shown how to detect and remove outliers using Z-score and IQR score.<jupyter_code>#Import the required libraries
import pandas as pd
from sklearn import datasets
from scipy import stats
import numpy as np
#Load the Boston House Pricing Dataset which is included in the sklearn dataset API
boston = datasets.load_boston()
x = boston.data
y = boston.target
columns = boston.feature_names
#Create the dataframe
boston_df = pd.DataFrame(boston.data)
boston_df.columns = columns
boston_df.head()<jupyter_output><empty_output><jupyter_text>Using Z-Score<jupyter_code>#Step1: Use Z-score function defined in scipy library to detect the outliers
boston_df_z = boston_df
z = np.abs(stats.zscore(boston_df))
print(z)<jupyter_output>[[0.41978194 0.28482986 1.2879095  ... 1.45900038 0.44105193 1.0755623 ]
 [0.41733926 0.48772236 0.59338101 ... 0.30309415 0.44105193 0.49243937]
 [0.41734159 0.48772236 0.59338101 ... 0.30309415 0.39642699 1.2087274 ]
 ...
 [0.41344658 0.48772236 0.11573841 ... 1.17646583 0.44105193 0.98304761]
 [0.40776407 0.48772236 0.11573841 ... 1.17646583 0.4032249  0.86530163]
 [0.41500016 0.48772236 0.11573841 ... 1.17646583 0.44105193 0.66905833]]<jupyter_text>Looking at the code and the output above, it is difficult to say which data point is an outlier.So let’s define a threshold to identify an outlier.<jupyter_code>#Step2: Define a threshold
threshold = 3
print(np.where(z > 3))<jupyter_output>(array([ 55,  56,  57, 102, 141, 142, 152, 154, 155, 160, 162, 163, 199,
       200, 201, 202, 203, 204, 208, 209, 210, 211, 212, 216, 218, 219,
       220, 221, 222, 225, 234, 236, 256, 257, 262, 269, 273, 274, 276,
       277, 282, 283, 283, 284, 347, 351, 352, 353, 353, 354, 355, 356,
       357, 358, 363, 364, 364, 365, 367, 369, 370, 372, 373, 374, 374,
       380, 398, 404, 405, 406, 410, 410, 411, 412, 412, 414, 414, 415,
       416, 418, 418, 419, 423, 424, 425, 426, 427, 427, 429, 431, 436,
       437, 438, 445, 450, 454, 455, 456, 457, 466], dtype=int64), array([ 1,  1,  1, 11, 12,  3,  3,  3,  3,  3,  3,  3,  1,  1,  1,  1,  1,
        1,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  5,  3,  3,  1,  5,
        5,  3,  3,  3,  3,  3,  3,  1,  3,  1,  1,  7,  7,  1,  7,  7,  7,
        3,  3,  3,  3,  3,  5,  5,  5,  3,  3,  3, 12,  5, 12,  0,  0,  0,
        0,  5,  0, 11, 11, 11, 12,  0, 12, 11, 11,  0, 11, 11, 11, 11, 11,
       11,  0, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11[...]<jupyter_text>The first array contains the list of row numbers and second array contains the respective column numbers, which means that z[55][1] has a z-score higher than 3.<jupyter_code>#Step3: Print the z-score of z[55][1]
print(z[55][1])<jupyter_output>3.375038763517309<jupyter_text>So, the data point — 55th record on column ZN is an outlier.<jupyter_code>#Step4: Remove the outliers using the z-score
boston_df_z = boston_df_z[(z < 3).all(axis=1)]

print("The no. of rows before outlier filtering was: ", boston_df.shape)
print("The no. of rows after outlier filtering is: ", boston_df_z.shape)<jupyter_output>The no. of rows before outlier filtering was:  (506, 13)
The no. of rows after outlier filtering is:  (415, 13)<jupyter_text>Hence, we filtered out around 90+ rows from the dataset i.e. outliers have been removed.  Using IQR Score<jupyter_code>#Step1: Calculate the IQR
boston_df_iqr = boston_df
Q1 = boston_df_iqr.quantile(0.25)
Q3 = boston_df_iqr.quantile(0.75)
IQR = Q3 - Q1
print(IQR)
#Step2: Detect the outliers
print(boston_df_iqr < (Q1 - 1.5 * IQR)) |(boston_df_iqr > (Q3 + 1.5 * IQR))<jupyter_output>CRIM     ZN  INDUS   CHAS    NOX     RM    AGE    DIS    RAD    TAX  \
0    False  False  False  False  False  False  False  False  False  False   
1    False  False  False  False  False  False  False  False  False  False   
2    False  False  False  False  False  False  False  False  False  False   
3    False  False  False  False  False  False  False  False  False  False   
4    False  False  False  False  False  False  False  False  False  False   
5    False  False  False  False  False  False  False  False  False  False   
6    False  False  False  False  False  False  False  False  False  False   
7    False  False  False  False  False  False  False  False  False  False   
8    False  False  False  False  False  False  False  False  False  False   
9    False  False  False  False  False  False  False  False  False  False   
10   False  False  False  False  False  False  False  False  False  False   
11   False  False  False  False  False  False  False  False  False  False   [...]<jupyter_text>The data point where we have False that means these values are valid whereas True indicates presence of an outlier.<jupyter_code>#Step3: Remove the outliers using the IQR score
boston_df_out = boston_df_iqr[~((boston_df_iqr < (Q1 - 1.5 * IQR)) |(boston_df_iqr > (Q3 + 1.5 * IQR))).any(axis=1)]

print("The no. of rows before outlier filtering was: ", boston_df_iqr.shape)
print("The no. of rows after outlier filtering is: ", boston_df_out.shape)<jupyter_output>The no. of rows before outlier filtering was:  (506, 13)
The no. of rows after outlier filtering is:  (274, 13)
{'fraction_non_alphanumeric': 0.10361126867082625, 'fraction_numerical': 0.15938740782756664, 'mean_word_length': 3.255832662912309, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 35, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '315674', 'license': 'MIT', 'n_tokens_mistral': 2648, 'n_tokens_neox': 2015, 'n_words': 709, 'path': 'm2_demo1 - Remove Outliers.ipynb', 'repo_name': 'MarvelAmazon/titanic-data-analyis', 'size': '5289'}