<jupyter_start><jupyter_text>Ways to Detect and Remove the OutliersWhile working on a Data Science project, what is it, that you look for? What is the most important part of the EDA phase? There are certain things which, if are not done in the EDA phase, can affect further statistical/Machine Learning modelling. One of them is finding “Outliers”. In this post we will try to understand what is an outlier? Why is it important to identify the outliers? What are the methods to outliers? Don’t worry, we won’t just go through the theory part but we will do some coding and plotting of the data too.Credit: https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba<jupyter_code>#Import the libraries
import numpy as np
import pandas as pd
from sklearn.datasets import load_boston,load_iris
#Load the data
boston = load_boston()

#Find features and target
x = boston.data
y = boston.target

#Find the dic keys
print(boston.keys())
#find features name
columns = boston.feature_names
columns
#Description of dataset
print(boston.DESCR)
#Create dataframe
boston_df = pd.DataFrame(boston.data)
boston_df.columns = columns
boston_df_o = boston_df
boston_df.shape
#Oulier detection - Univarite - Boxplot
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
sns.boxplot(x=boston_df['DIS'])
#Check the correlation between features before multivariate outlier analysis
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

plt.figure(figsize= (10,10), dpi=100)
sns.heatmap(boston_df.corr())
#Multivariate outlier analysis
fig, ax = plt.subplots(figsize=(16,8))
ax.scatter(boston_df['INDUS'], boston_df['TAX'])
ax.set_xlabel('Proportion of non-retail business acres per town')
ax.set_ylabel('Full-value property-tax rate per $10,000')
plt.show()
from scipy import stats
import numpy as np
z = np.abs(stats.zscore(boston_df))
print(z)
z.shape
threshold = 3
print(np.where(z > 3))
#print(boston_df[np.where(z > 3)])
print(z[55][1])<jupyter_output><empty_output><jupyter_text>Removing Outliers<jupyter_code>boston_df_o = boston_df_o[(z < 3).all(axis=1)]
boston_df.shape
boston_df_o.shape
boston_df_o1 = boston_df
Q1 = boston_df_o1.quantile(0.25)
Q3 = boston_df_o1.quantile(0.75)
IQR = Q3 - Q1
print(IQR)

boston_df_out = boston_df_o1[~((boston_df_o1 < (Q1 - 1.5 * IQR)) |(boston_df_o1 > (Q3 + 1.5 * IQR))).any(axis=1)]
boston_df_out.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10004133939644481, 'fraction_numerical': 0.02232327408019843, 'mean_word_length': 5.914285714285715, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '534289', 'license': 'MIT', 'n_tokens_mistral': 920, 'n_tokens_neox': 865, 'n_words': 258, 'path': 'code/week2_dataprep/Lab4_OutliersWithLog_Boston.ipynb', 'repo_name': 'stangthanin/2110446_DataScience_2021s2', 'size': '2419'}