<jupyter_start><jupyter_text>Discussion Related With Outliers And Impact On Machine Learning!! What is an outlier ?An outlier is an observation that lies an abnormal distance from other values in a random sample from a population Which Machine LEarning Models Are Sensitive To Outliers?1. Naivye Bayes Classifier-------------------Not Sensitive To Outliers2. SVM --------------------------------------Not Sensitive To Outliers          3. Linear Regression-------------------------Sensitive To Outliers4. Logistic Regression-----------------------Sensitive To Outliers5. Decision Tree Regressor or Classifier---- Not Sensitive6. Ensemble(RF,XGboost,GB)-------------------Not Sensitive7. KNN---------------------------------------Not Sensitive 8. Kmeans------------------------------------Sensitive9. Hierarichal-------------------------------Sensitive 10. PCA--------------------------------------Sensitive 11. Neural Networks--------------------------Sensitive12. DBScan-----------------------------------Sensitive<jupyter_code>import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import seaborn as sns

%matplotlib inline
df = pd.read_csv('titanic.csv')
df.head()
df.isnull().sum()
sns.displot(df['Age'].dropna(),kde=True)
plt.show()
sns.displot(df['Age'].fillna(100),kde=True)
plt.show()<jupyter_output><empty_output><jupyter_text>Gaussian Distributed<jupyter_code>sns.histplot(df['Age'], bins=50)

plt.title('Age')
plt.xlabel('Age')
plt.ylabel('No of passenger')

plt.show()
sns.boxplot(x=df["Age"])
df['Age'].describe()
df_copy = df.copy()<jupyter_output><empty_output><jupyter_text>If The Data Is Normally Distributed or close to<jupyter_code>##### Assuming Age follows A Gaussian Distribution we will calculate the boundaries which differentiates the outliers

uppper_boundary=df['Age'].mean() + 3* df['Age'].std()
lower_boundary=df['Age'].mean() - 3* df['Age'].std()
print(lower_boundary)
print(uppper_boundary)
print(df['Age'].mean())
filt_left = df_copy['Age'] <= lower_boundary
filt_right = df_copy['Age'] >= uppper_boundary

df_copy.loc[filt_left ,'Age'] = lower_boundary
df_copy.loc[filt_right ,'Age'] = uppper_boundary
plt.figure(figsize=(10, 6))

plt.subplot(1, 2, 1)
sns.histplot(df['Age'], bins=50)
plt.title('before')

plt.subplot(1, 2, 2)
sns.histplot(df_copy['Age'], bins=50)
plt.title('after')

plt.show()
plt.figure(figsize=(10, 6))

plt.subplot(1, 2, 1)
sns.distplot(x=df['Age'], kde=True)
plt.title('before')

plt.subplot(1, 2, 2)
sns.distplot(x=df_copy['Age'], kde=True)
plt.title('after')

plt.show()
plt.figure(figsize=(10, 6))

plt.subplot(1, 2, 1)
sns.boxplot(x=df['Age'])
plt.title('before')

plt.subplot(1, 2, 2)
sns.boxplot(x=df_copy['Age'])
plt.title('after')

plt.show()
compare = pd.DataFrame()
compare
compare = pd.DataFrame()
compare['index'] = df['Age'].describe().index

compare['before'] = df['Age'].describe().values
compare['after'] = df_copy['Age'].describe().values

compare<jupyter_output><empty_output><jupyter_text>If data is Skewed (not normal)<jupyter_code>figure=df.Fare.hist(bins=50)
figure.set_title('Fare')
figure.set_xlabel('Fare')
figure.set_ylabel('No of passenger')
df.boxplot(column="Fare")
df['Fare'].describe()
#### Lets compute the Interquantile range to calculate the boundaries
IQR=df.Fare.quantile(0.75)-df.Fare.quantile(0.25)
lower_bridge=df['Fare'].quantile(0.25)-(IQR*1.5)
upper_bridge=df['Fare'].quantile(0.75)+(IQR*1.5)

print(lower_bridge)
print(upper_bridge)
#### Extreme outliers
lower_bridge = df['Fare'].quantile(0.25)-(IQR*3)
upper_bridge = df['Fare'].quantile(0.75)+(IQR*3)
print(lower_bridge), print(upper_bridge)<jupyter_output>-61.358399999999996
100.2688<jupyter_text>here if we use strict boundary meaning 1.5 then for our data it will not cover all. First try loose one if still a lot of outliers are still present then be tight.<jupyter_code>filt_left = df_copy['Fare'] <= lower_bridge
filt_right = df_copy['Fare'] >= upper_bridge

df_copy.loc[filt_left ,'Fare'] = lower_bridge
df_copy.loc[filt_right ,'Fare'] = upper_bridge
plt.figure(figsize=(10, 6))

plt.subplot(1, 2, 1)
sns.histplot(df['Fare'], bins=50)
plt.title('before')

plt.subplot(1, 2, 2)
sns.histplot(df_copy['Fare'], bins=50)
plt.title('after')

plt.show()
plt.figure(figsize=(10, 6))

plt.subplot(1, 2, 1)
sns.distplot(x=df['Fare'], kde=True)
plt.title('before')

plt.subplot(1, 2, 2)
sns.distplot(x=df_copy['Fare'], kde=True)
plt.title('after')

plt.show()
plt.figure(figsize=(10, 6))

plt.subplot(1, 2, 1)
sns.boxplot(x=df['Fare'])
plt.title('before')

plt.subplot(1, 2, 2)
sns.boxplot(x=df_copy['Fare'])
plt.title('after')

plt.show()
compare = pd.DataFrame()
compare['index'] = df['Fare'].describe().index

compare['before'] = df['Fare'].describe().values
compare['after'] = df_copy['Fare'].describe().values

compare<jupyter_output><empty_output><jupyter_text>check difference  logistic regression<jupyter_code>from sklearn.model_selection import train_test_split


X_train,X_test,y_train,y_test=train_test_split(df[['Age','Fare']].fillna(0),df['Survived'],test_size=0.3)

X_train_copy,X_test_copy,y_train_copy,y_test_copy=train_test_split(df_copy[['Age','Fare']].fillna(0),df_copy['Survived'],test_size=0.3)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score,roc_auc_score


clf1 = LogisticRegression()
clf2 = LogisticRegression()
clf1.fit(X_train, y_train)
clf2.fit(X_train_copy, y_train_copy)
y_pred1 = clf1.predict(X_test)
y_pred2 = clf1.predict_proba(X_test)

y_pred_copy1 = clf2.predict(X_test_copy)
y_pred_copy2 = clf2.predict_proba(X_test_copy)
print('='*50)
print('Before')

print("Accuracy_score: {}".format(accuracy_score(y_test ,y_pred1)))

print("roc_auc_score: {}".format(roc_auc_score(y_test, y_pred2[:,1])))

print('='*50)
print('After')

print("Accuracy_score: {}".format(accuracy_score(y_test_copy, y_pred_copy1)))

print("roc_auc_score: {}".format(roc_auc_score(y_test_copy, y_pred_copy2[:,1])))<jupyter_output>==================================================
Before
Accuracy_score: 0.6791044776119403
roc_auc_score: 0.701156734496124
==================================================
After
Accuracy_score: 0.6343283582089553
roc_auc_score: 0.617671996124031<jupyter_text>random forest<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score,roc_auc_score


clf1 = RandomForestClassifier()
clf2 = RandomForestClassifier()
clf1.fit(X_train, y_train)
clf2.fit(X_train_copy, y_train_copy)
y_pred1 = clf1.predict(X_test)
y_pred2 = clf1.predict_proba(X_test)

y_pred_copy1 = clf2.predict(X_test_copy)
y_pred_copy2 = clf2.predict_proba(X_test_copy)
print('='*50)
print('Before')

print("Accuracy_score: {}".format(accuracy_score(y_test ,y_pred1)))

print("roc_auc_score: {}".format(roc_auc_score(y_test, y_pred2[:,1])))

print('='*50)
print('After')

print("Accuracy_score: {}".format(accuracy_score(y_test_copy, y_pred_copy1)))

print("roc_auc_score: {}".format(roc_auc_score(y_test_copy, y_pred_copy2[:,1])))<jupyter_output>==================================================
Before
Accuracy_score: 0.7164179104477612
roc_auc_score: 0.7287124515503876
==================================================
After
Accuracy_score: 0.6567164179104478
roc_auc_score: 0.7002785852713178
{'fraction_non_alphanumeric': 0.22075419763281035, 'fraction_numerical': 0.04404073768235618, 'mean_word_length': 5.829887218045113, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '192317', 'license': 'MIT', 'n_tokens_mistral': 2967, 'n_tokens_neox': 2646, 'n_words': 457, 'path': 'techniques with code/Removing Outliers from dataset.ipynb', 'repo_name': 'ikaushikpal/Feature-Engineering-ML', 'size': '7266'}