<jupyter_start><jupyter_text>Discussion Related With Outliers And Impact On Machine Learning!!  Which Machine LEarning Models Are Sensitive To Outliers?1.  Naivye Bayes Classifier -------- Not Sensitive To Outliers2.  SVM -------- Not Sensitive To Outliers3.  Linear Regression ---------- Sensitive To Outliers4.  Logistic Regression ------- Sensitive To Outliers5.  Decision Tree Regressor or Classifier ---- Not Sensitive6.  Ensemble(RF,XGboost,GB) ------- Not Sensitive7.  KNN --------------------------- Not Sensitive8.  Kmeans ------------------------ Sensitive9.  Hierarichal ------------------- Sensitive10. PCA -------------------------- Sensitive11. Neural Networks -------------- Sensitive<jupyter_code>df = pd.read_csv("data/titanic.csv")
df.head()
df.isnull().sum()
# Check type of distribution and outliers during dropna()

sns.displot(df['Age'].dropna())

# it is a normal distribution 
# Check type of distribution and outliers during fillna() 

sns.displot(df['Age'].fillna(100))
# Check if there are outliers without affecting NaN values (no dropna() or fillna())
figure=df.Age.hist(bins=50)
figure.set_title('Age')
figure.set_xlabel('Age')
figure.set_ylabel('No of passenger')

# distribution is normal/guassian distribution
# Identify the Outliers 

figure = df.boxplot(column="Age")

# The points in the box plot (from 65 to 80) --> outliers
df['Age'].describe()<jupyter_output><empty_output><jupyter_text>1) Technique: If The Data Is Normally/Gaussian Distributed, We use this Assuming Age follows A Gaussian Distribution we will **calculate the boundaries which differentiates the outliers**<jupyter_code>upper_boundary = df['Age'].mean() + 3 * df['Age'].std() # 3rd standard deviation
lower_boundary = df['Age'].mean() - 3 * df['Age'].std()
print(lower_boundary), print(upper_boundary), print(df['Age'].mean())

# in Age, -ve value will not be there
### Lets compute the interquartile range to calculate the boundaries 

# IQR = 75th percentile - 25th percentile
IQR = df['Age'].quantile(0.75) - df['Age'].quantile(0.25)
IQR

# Any point outside the IQR range is identified as an outlier
# from 25% percentile, subtract IQR -- 1.5 to the left
# from 75% percentile, add IQR -- 1.5 to the right

lower_bridge = df['Age'].quantile(0.25)-(IQR * 1.5)
upper_bridge = df['Age'].quantile(0.75)+(IQR * 1.5)
print(lower_bridge), print(upper_bridge)

# For Extreme outliers --> IQR*3<jupyter_output>-6.6875
64.8125<jupyter_text>**Conclusion:** max value was 80.0. After deduction, anything above 64.8125 is an outlier  2) Technique: If Features are Skewed, We use this<jupyter_code># Check if there are outliers without affecting NaN values (no dropna() or fillna())

figure=df.Age.hist(bins=50)
figure.set_title('Fair')
figure.set_xlabel('Fair')
figure.set_ylabel('No of passenger')

# Dataset over here is Skewed
# Identify the Outliers 

figure = df.boxplot(column="Fare")

# The points in the box plot --> outliers (Huge amount of outliers)
df['Fare'].describe()

# difference between 75% percentile and max is huge, so there are a lot of outliers
### Lets compute the Interquantile range to calculate the boundaries
IQR = df['Fare'].quantile(0.75) - df['Fare'].quantile(0.25)
IQR
# For skewed dataset - both left skewed and right skewed, we will consider extreme outliers

lower_bridge = df['Fare'].quantile(0.25)-(IQR * 3)
upper_bridge = df['Fare'].quantile(0.75)+(IQR * 3)
print(lower_bridge), print(upper_bridge)

# we are getting 100, this is fairly good<jupyter_output>-61.358399999999996
100.2688<jupyter_text>Now that we have gotten the outliers, how do we remove them from the dataset? * before applying the dataset to a ML algorithm<jupyter_code>new_df = df.copy()
# we know the outlier of 'Age' is 73.27860964406094

# if 'Age' amount is >=73, with respect to 'Age' feature, we will replace all outliers above 73.278 with 73
new_df.loc[new_df['Age']>=73,'Age'] = 73
# we know the outlier of 'Fair' is 100.2688

# if 'Fare' amount is >=100, with respect to 'Fare' feature, we will replace all outliers above 100.2688 with 100
new_df.loc[new_df['Fare']>=100,'Fare'] = 100
# Now lets see the new dataset

# Age

figure=new_df.Age.hist(bins=50)
figure.set_title('Age')
figure.set_xlabel('Age')
figure.set_ylabel('No of passenger')

# Most points are 73
# Fare

figure=new_df.Fare.hist(bins=50)
figure.set_title('Fare')
figure.set_xlabel('Fare')
figure.set_ylabel('No of passenger')

# Most points are 100
# For now we will apply on a ml algorithm

from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(new_df[['Age','Fare']].fillna(0),new_df['Survived'],test_size=0.3) 

# Feature Engineering applied as well
### Logistic Regression --> sensitive to outliers

from sklearn.linear_model import LogisticRegression
classifier=LogisticRegression()
classifier.fit(X_train,y_train) # Train model

# Test model
y_pred=classifier.predict(X_test)
y_pred1=classifier.predict_proba(X_test)

# Evaluation metrics
from sklearn.metrics import accuracy_score,roc_auc_score
print("Accuracy_score: {}".format(accuracy_score(y_test,y_pred)))
print("roc_auc_score: {}".format(roc_auc_score(y_test,y_pred1[:,1]))) # take all rows, and first column
# The format() method formats the specified value(s) and insert them inside the string's placeholder.

# The placeholder is defined using curly brackets: {}

# The format() method returns the formatted string.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14199006805223469, 'fraction_numerical': 0.03512966709582491, 'mean_word_length': 4.706190975865687, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '168480', 'license': 'MIT', 'n_tokens_mistral': 1972, 'n_tokens_neox': 1703, 'n_words': 543, 'path': 'Outliers/Outliers.ipynb', 'repo_name': 'Jawwad-Fida/Feature-Engineering', 'size': '5437'}