<jupyter_start><jupyter_text>IBM: Feature Engineering and Bias Detection Class ImbalanceIf one or more classes in your data are underrepresented you could be introducing bias into your models. Working with imbalanced data may also lead your analysis to incorrect or misleading conclusions. Understanding this will help you keep class balance and bias in perspective as you proceed through the data analysis pipeline.Dealing with problems of class imbalance often requires finesse, and this unit introduces several potential approaches. These methods employ different tactics to avoid being overwhelmed by the majority class. Typically being overwhelmed during the model’s training directly affects the model’s accuracy. Accuracy, that is the proportion of predictions that are correct, is an intuitive metric for classification, but can be misleading with imbalanced classes. For example, if one percent of a processor’s transactions in a training sample are fraudulent, then a model that always predicts “not fraudulent” will be 99% accurate but of no practical use. WARNING: Accuracy is not an appropriate metric for imbalanced classes. Use F1 Score (harmonic mean) or another metric instead.The solution to misleading metrics is to break the correct vs. incorrect dichotomy of binary classification into four pieces: true positives, true negatives, false positives, and false negatives. There are many ways to combine these values with an even larger number of naming conventions.   The solution to imbalanced classes is to up-sample or down-sample data, making the minority class more common through some type of repetition or making the majority class less common by throwing away some observations. Up-sampling & down-samplingThe most common approaches are sampling based, or more specifically, re-sampling based. Between up-sampling and down-sampling (also called over-sampling and under-sampling), down-sampling is a bit simpler conceptually. Fundamentally, if you have a minority class or classes that is noticeably underrepresented. In a random way you can drop some of those from the training data so that the proportions are more closely matched across classes. There are additional methods, one of which is inspired by K Nearest Neighbors (KNN) that can improve on pure random selection. A major caveat to down-sampling is that we are not using all of our data. Over-sampling techniques also come in several flavors, from random or naive versions to classes of algorithms like the Synthetic Minority Oversampling Technique (SMOTE) and the Adaptive Synthetic (ADASYN) sampling method. Synthetic Minority Over-sampling Technique for Nominal and Continuous (SMOTENC) is an extension of the original SMOTE method designed to handle a mixture of categorical and continuous features. SMOTE has a number of other variants including ones that make use of Support Vector Machines and K-means clustering to improve on the synthetic samples.<jupyter_code># import essentials
import numpy as np
import pandas as pd
import seaborn as sns

from sklearn.datasets import load_boston

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.feature_selection import SelectKBest
from sklearn.metrics import median_absolute_error, r2_score

import plotly.express as px  # interactive data visualization

sns.set()
%matplotlib inline
# quick example of StandardScaler()
# quick example of Pipelines

boston = load_boston()                                                                                   ## load the boston dataset
X, y = boston['data'], boston['target']
features = boston['feature_names']

## split the data to a training set and a test set
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

## create a pipeline
pipe = make_pipeline((StandardScaler()),
                     (SelectKBest(k=10)),
                     (RandomForestRegressor(n_estimators=20)))

## train on the training data
pipe.fit(X_train, y_train)

## evaluate the model with the test data
y_pred = pipe.predict(X_test)
print(r'R^2=%.2f, MAE=%.2f'%(r2_score(y_test, y_pred), median_absolute_error(y_test, y_pred)))<jupyter_output>R^2=0.76, MAE=1.35<jupyter_text>Case-study- Model Sensitivity to Class Imbalance: some models are more or less sensitive to class imbalance- Sampling Techniques: up-sampling and down-sampling- Imbalanced-learn: introduce the package and use SMOTE as an example<jupyter_code>imbalanced = pd.read_csv('datasets/aavail-target.csv')
print(imbalanced.head(3))
imbalanced.info()
fig = px.histogram(imbalanced, x='is_subscriber')
fig.show()

fig = px.histogram(imbalanced, x='subscriber_type')
fig.show()
imbalanced[imbalanced.is_subscriber == 0]['subscriber_type']

# pull out the target and remove unnecessary columns
_y = imbalanced.pop('is_subscriber')
y = np.zeros(_y.size)
y[_y == 0] = 1
imbalanced.drop(columns=['customer_id', 'customer_name'], inplace=True)

# split the data
X_train, X_test, y_train, y_test = train_test_split(imbalanced, y, test_size=.2, stratify=y)
print('Train Set:\n', pd.DataFrame(y_train, columns=['vals']).value_counts())
print('\n')
print('Test Set:\n', pd.DataFrame(y_test, columns=['vals']).value_counts())
from collections import Counter
print(sorted(Counter(y_train).items()))
print(sorted(Counter(y_test).items()))


# transformation pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer 
from sklearn.pipeline import Pipeline 

from sklearn.linear_model import LogisticRegression

numeric_features = ['age', 'num_streams']
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_features = ['country', 'subscriber_type']
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ]
)

# model pipeline
clf = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression(solver='lbfgs'))
])

from sklearn.metrics import classification_report 

clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)

print(classification_report(y_test, y_pred, target_names=['subscriber', 'churn']))<jupyter_output>precision    recall  f1-score   support

  subscriber       0.83      0.87      0.85       142
       churn       0.63      0.57      0.60        58

    accuracy                           0.78       200
   macro avg       0.73      0.72      0.72       200
weighted avg       0.77      0.78      0.78       200<jupyter_text>Imbalanced Learn- Simple interface & integrations with sklearn- Neural Networks: imblearn.keras & imblearn.tensorflow- SMOTE & related variants- Works naturally with pipelines<jupyter_code># !pip install -U imbalanced-learn
# example with toy dataset
from sklearn.datasets import make_classification
from imblearn.over_sampling import RandomOverSampler

# instantiate data
X, y = make_classification(n_samples=5000, n_features=2, n_informative=2,
                          n_redundant=0, n_repeated=0, n_classes=3,
                          n_clusters_per_class=1, weights=[.01, .05, .94],
                          class_sep=.8, random_state=0)
print('Original Target')
print(sorted(Counter(y).items()))
print(f'X shape: {X.shape}\ny shape: {y.shape}')
X_resampled, y_resampled = RandomOverSampler(random_state=0).fit_resample(X, y)
print('Resampled Target')
print(sorted(Counter(y_resampled).items()))
print(f'Resampled X shape: {X_resampled.shape}\nResampled y shape: {y_resampled.shape}')<jupyter_output>Resampled Target
[(0, 4674), (1, 4674), (2, 4674)]
Resampled X shape: (14022, 2)
Resampled y shape: (14022,)<jupyter_text>Class Imbalance Complete Pipeline<jupyter_code>import imblearn.pipeline as pl
from imblearn.pipeline import make_pipeline
from imblearn.over_sampling import RandomOverSampler, SMOTE
# ORIGINAL PIPELINE
clf1 = pl.Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression(solver='lbfgs'))
])

clf2 = pl.Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('smote', RandomOverSampler(random_state=42)),
    ('classifier', LogisticRegression(solver='lbfgs'))
])

clf3 = pl.Pipeline([
    ('preprocessor', preprocessor),
    ('smote', SMOTE(random_state=42)),
    ('classifier', LogisticRegression(solver='lbfgs'))
])

for clf in [clf1, clf2, clf3]:
    clf.fit(X_train, y_train)
for name, clf in [('no sampling', clf1), ('random oversampling', clf2), ('SMOTE', clf3)]:
    y_pred = clf.predict(X_test)
    print(name)
    print(classification_report(y_test, y_pred, target_names=['subscriber', 'churn']))<jupyter_output>no sampling
              precision    recall  f1-score   support

  subscriber       0.83      0.87      0.85       142
       churn       0.63      0.57      0.60        58

    accuracy                           0.78       200
   macro avg       0.73      0.72      0.72       200
weighted avg       0.77      0.78      0.78       200

random oversampling
              precision    recall  f1-score   support

  subscriber       0.83      0.82      0.83       142
       churn       0.57      0.60      0.59        58

    accuracy                           0.76       200
   macro avg       0.70      0.71      0.71       200
weighted avg       0.76      0.76      0.76       200

SMOTE
              precision    recall  f1-score   support

  subscriber       0.83      0.82      0.83       142
       churn       0.57      0.60      0.59        58

    accuracy                           0.76       200
   macro avg       0.70      0.71      0.71       200
weighted avg       0.76      0.76   [...]
{'fraction_non_alphanumeric': 0.08467138725441309, 'fraction_numerical': 0.02922110302184103, 'mean_word_length': 3.6382978723404253, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '401106', 'license': 'MIT', 'n_tokens_mistral': 3106, 'n_tokens_neox': 2887, 'n_words': 980, 'path': 'datacamp_ml/ml_classification_regression/ibm_class_imbalance.ipynb', 'repo_name': 'issagaliyeva/machine_learning', 'size': '10027'}