<jupyter_start><jupyter_text>Read and understand data<jupyter_code>data = pd.read_csv('creditcard.csv')
print(data.shape)
data.head()<jupyter_output>(284807, 31)<jupyter_text>Load the data and reduce the memory usage of each dataframe<jupyter_code>def reduce_mem_usage(df):
    """ iterate through all the columns of a dataframe and modify the data type
        to reduce memory usage.        
    """
    start_mem = df.memory_usage().sum() / 1024**2
    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))
    
    for col in df.columns:
        col_type = df[col].dtype
        
        if col_type != object:
            c_min = df[col].min()
            c_max = df[col].max()
            if str(col_type)[:3] == 'int':
                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                    df[col] = df[col].astype(np.int8)
                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                    df[col] = df[col].astype(np.int16)
                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                    df[col] = df[col].astype(np.int32)
                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                    df[col] = df[col].astype(np.int64)  
            else:
                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                    df[col] = df[col].astype(np.float16)
                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                    df[col] = df[col].astype(np.float32)
                else:
                    df[col] = df[col].astype(np.float64)
        else:
            df[col] = df[col].astype('category')

    end_mem = df.memory_usage().sum() / 1024**2
    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))
    
    return df

data = reduce_mem_usage(data)
data.dtypes
object_types = data.select_dtypes(include=['object','category'])
object_types.columns<jupyter_output><empty_output><jupyter_text>Statistical Summary<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>Summarizing the number of unique values in each column<jupyter_code>print(data.nunique())<jupyter_output>Time      124592
V1         17239
V2         21462
V3         18221
V4         18661
V5         20454
V6         19734
V7         21084
V8         23301
V9         19442
V10        21042
V11        17517
V12        19264
V13        18406
V14        20199
V15        18066
V16        18899
V17        19236
V18        18716
V19        19281
V20        22820
V21        22264
V22        18352
V23        22698
V24        19392
V25        18426
V26        17050
V27        23887
V28        23023
Amount      8976
Class          2
dtype: int64<jupyter_text>Columns with  single value Columns that have a single value are probably useless for modeling. Thesecolumns are referred to zero-variance predictors, so we dont need to use them anymore.<jupyter_code>values = data.nunique()
single_value_cols = [values.index[i] for i,v in enumerate(values) if v == 1]
single_value_cols<jupyter_output><empty_output><jupyter_text>Checking duplicate rowsDuplicate row is a row where each value in each column for that row appears in identically the same column values in another row.  They are could be useless to the modeling process. duplicated() function will report whether a given row is duplicated or not. Allrows are marked as either False to indicate that it is not a duplicate or True to indicate thatit is a duplicate.<jupyter_code>dups = data.duplicated()
print(dups.any())
data['Class'].value_counts()
data.drop_duplicates(inplace=True)
print(data.shape)
data['Class'].value_counts()<jupyter_output><empty_output><jupyter_text>Explotary Data Analysis<jupyter_code>f,ax=plt.subplots(1,2,figsize=(18,8))
data['Class'].value_counts().plot.pie(explode=[0,0.1],autopct='%1.1f%%',ax=ax[0],shadow=True)
ax[0].set_title('Class')
ax[0].set_ylabel('')
sns.countplot('Class',data=data,ax=ax[1])
ax[1].set_title('Class')
plt.show()<jupyter_output><empty_output><jupyter_text>The graphs show that our dataset is highly imbalanced.First let us understand the different types of features.  Distribution  of anomaly and normal  data<jupyter_code>def plot_histogram(df, bins, column, log_scale=False):
    bins = 100
    anomalies = df[df.Class == 1]
    normal = df[df.Class == 0]
    fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
    fig.suptitle(f'Counts of {column} by Class')
    ax1.hist(anomalies[column], bins = bins, color="red")
    ax1.set_title('Anomaly')
    ax2.hist(normal[column], bins = bins, color="orange")
    ax2.set_title('Normal')
    plt.xlabel(f'{column}')
    plt.ylabel('Count')
    if log_scale:
        plt.yscale('log')
    plt.xlim((np.min(df[column]), np.max(df[column])))
    plt.show()<jupyter_output><empty_output><jupyter_text>A histogram of data for the column V12 in df. As wecan see, there is a very clear deviation seen with the anomalous valuescompared to the normal values. Both plots share the same x-axisscale, so while the counts might be very low compared to the normalvalues, they are still spread out far more than the normal values forthe same range of V12 column values.<jupyter_code>for f in range(1, 29):
    print(f'V{f} Counts')
    plot_histogram(data,100, f'V{f}')<jupyter_output>V1 Counts<jupyter_text>From this, you can definitely notice a right skew as well as the massive outliers present in the normal data. Other than a few defined spikes that stand out from where the normal points would have been, most of the fraudulent data in this context seems to blend in with the normal data.  Distrubiton of all numeric features<jupyter_code>numeric = data.select_dtypes(include = 'number')
for col in list(numeric)[1:]:
    plt.figure(figsize=(12,5))
    plt.title(f"Distribution of {col}")
    ax = sns.distplot(data[col])
    plt.show()<jupyter_output><empty_output><jupyter_text>As we see, we have huge amount of outliers which have to be handled before modeling.  Data Splitting<jupyter_code>def load_data(data_path):
    data = pd.read_csv(data_path)
    data = reduce_mem_usage(data)
    X = data.iloc[:,:-1]
    y = data.iloc[:,-1]
    
    X_train,X_test,Y_train,Y_test = train_test_split(X,y,test_size=0.3, shuffle = True)
    
    return X_train,X_test,Y_train,Y_test

X_train,X_test,Y_train,Y_test = load_data('creditcard.csv')<jupyter_output>Memory usage of dataframe is 67.36 MB
Memory usage after optimization is: 17.11 MB
Decreased by 74.6%<jupyter_text>Evaluation Firstly, because most of the standardmetrics that are widely used assume a balanced class distribution, and because typically notall classes, and therefore, not all prediction errors, are equal for imbalanced classification. So, accuracy is inappropriate forimbalanced classification problems. The main reason is that the overwhelming number ofexamples from the majority class (or classes) will overwhelm the number of examples in theminority class, meaning that even unskillful models can achieve accuracy scores of 90 percent,or 99 percent, depending on how severe the class imbalance happens to be. As a evaluation metric I am using __balanced_accuracy_score, roc_auc_score, geometric_mean_score.__<jupyter_code>def evaluate_model(ytrue, ypred):
    metrics = {}
    metrics['balanced_accuracy_score']=balanced_accuracy_score(ytrue, ypred)
    metrics['roc_auc_score'] = roc_auc_score(ytrue, ypred)
    metrics['geometric_mean_score'] = geometric_mean_score(ytrue, ypred)
    return metrics<jupyter_output><empty_output><jupyter_text>Trying different approaches Inthis project, I have discovered metrics that we can use for imbalanced classification. Beside of this, I have used different approaches: 1. Data Sampling MethodsData sampling provides a collection of techniques that transform a training dataset inorder to balance or better balance the class distribution. lthough often described in terms of two-class classification problems, class imbalance alsoaffects those datasets with more than two classes that may have multiple minority classesor multiple majority classes. __Oversampling methods duplicate examples in the minority class or synthesize new examplesfrom the examples in the minority class. Meanwhile, Undersampling methods delete or select a subset of examples from the majority class.__  1.1 SMOTE (Synthetic Minority Oversampling Technique)SMOTE first selects a minority class instance a at random and finds its k nearestminority class neighbors. The synthetic instance is then created by choosing one ofthe k nearest neighbors b at random and connecting a and b to form a line segmentin the feature space. The synthetic instances are generated as a convex combinationof the two chosen instances a and b. 1.2  ADASYN (Adaptive Synthetic Sampling)It is a modification of SMOTE that is based on the idea of adaptively generating minority data samplesaccording to their distributions: more synthetic data is generated for minority classsamples that are harder to learn compared to those minority samples that are easierto learn. 1.3 Borderline-SMOTEA popular extension to SMOTE involves selecting those instances of the minority class that aremisclassified, such as with a k-nearest neighbor classification model. We can then oversamplejust those difficult instances, providing more resolution only where it may be required. 1.4 SVM-SMOTEThis method is an alternative to Borderline-SMOTE where a SVM algorithmis used instead of a KNN to identify misclassified examples on the decision boundary. In the SVMSMOTE(), borderline area is approximated by the support vectors obtained after traininga standard SVMs classifier on the original training set. New instances will berandomly created along the lines joining each minority class support vector with anumber of its nearest neighbors using the interpolation 1.5 Near Miss UndersamplingIn this method, we have three versions ofthe technique, named NearMiss-1, NearMiss-2, and NearMiss-3. Here, distance is determined in feature spaceusing Euclidean distance or similar. NearMiss-1: Majority class examples with minimum average distance to three closestminority class examples. NearMiss-2: Majority class examples with minimum average distance to three furthestminority class examples. NearMiss-3: Majority class examples with minimum distance to each minority classexample.<jupyter_code>names = [SMOTE(),NearMiss(),BorderlineSMOTE(),SVMSMOTE(),ADASYN()]
names
def sampling_method(method_name,xtrain,ytrain):
    resample = method_name
    model = LogisticRegression()
    X_train_res,Y_train_res=resample.fit_resample(xtrain,ytrain)
    start_time = time.time()
    print(X_train_res.shape,Y_train_res.shape)
    model.fit(X_train_res,Y_train_res)
    pred = model.predict(X_test)
    
    metrics_dict = evaluate_model(Y_test, pred)
    metrics_dict['model_name'] = str(model)
    metrics_dict['resample_method'] = str(resample)
    metrics_dict['train_time']= time.time() - start_time
    return metrics_dict
resample_results = []
for i in tqdm(names):
    result = sampling_method(i,X_train,Y_train)
    resample_results.append(result)<jupyter_output>0%|                                                                                            | 0/5 [00:00<?, ?it/s]<jupyter_text>Probalistic modelsProbabilistic models arethose models that are fit on the data under a probabilistic framework and often perform wellin general for imbalanced classification dataset. I will evaluate a suite of models that are known to be effective at predicting probabilities.Specifically, these are models that are fit under a probabilistic framework and explicitly predict acalibrated probability for each example. A such, this makes them well-suited to this dataset, evenwith the class imbalance. We will evaluate the following six probabilistic models implementedwith the scikit-learn library: Logistic Regression (LR) Linear Discriminant Analysis (LDA) Quadratic Discriminant Analysis (QDA) Gaussian Naive Bayes (GNB) Gaussian Process (GPC)<jupyter_code>prob_models = [LinearDiscriminantAnalysis(), LogisticRegression(solver='lbfgs'), QuadraticDiscriminantAnalysis(), GaussianNB(), GaussianProcessClassifier()]
def prob_method(model_name,xtrain,ytrain):
    model = model_name
    start_time = time.time()
    print(xtrain.shape,ytrain.shape)
    model.fit(xtrain, ytrain)
    pred = model.predict(X_test)
    
    metrics_dict = evaluate_model(Y_test, pred)
    metrics_dict['model_name'] = str(model)
    metrics_dict['resample_method'] = 'Probalistic'
    metrics_dict['train_time']= time.time() - start_time
    return metrics_dict
prob_results = []

for i in tqdm(prob_models):
    result = prob_method(i, X_train, Y_train)
    prob_results.append(result)<jupyter_output>0%|                                                                                            | 0/5 [00:00<?, ?it/s]<jupyter_text>Cost sensitive modelsSome machine learning algorithms can be adapted to pay more attention to one class thananother when fitting the model. These are referred to as cost-sensitive machine learningmodels and they can be used for imbalanced classification by specifying a cost that is inverselyproportional to the class distribution.Cost-sensitive learning is a subfield of machine learning that takes the costs of predictionerrors (and potentially other costs) into account when training a machine learning model. It isa field of study that is closely related to the field of imbalanced learning that is concerned withclassification on datasets with a skewed class distribution.<jupyter_code>cost_sens_models = [LogisticRegression(solver='lbfgs', class_weight='balanced'), DecisionTreeClassifier(class_weight='balanced')]
def cost_method(model_name,xtrain,ytrain):
    model = model_name
    start_time = time.time()
    print(xtrain.shape,ytrain.shape)
    model.fit(xtrain, ytrain)
    pred = model.predict(X_test)
    
    metrics_dict = evaluate_model(Y_test, pred)
    metrics_dict['model_name'] = str(model)
    metrics_dict['resample_method'] = 'Cost Sensitive'
    metrics_dict['train_time']= time.time() - start_time
    return metrics_dict
cost_results = []

for i in tqdm(cost_sens_models):
    result = cost_method(i, X_train, Y_train)
    cost_results.append(result)
results = resample_results + prob_results + cost_results
final = pd.DataFrame(results)
final
final.sort_values(by = 'balanced_accuracy_score', ascending=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08210180623973727, 'fraction_numerical': 0.024220032840722498, 'mean_word_length': 4.064795564795565, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '498046', 'license': 'MIT', 'n_tokens_mistral': 4568, 'n_tokens_neox': 4241, 'n_words': 1418, 'path': 'imbalanced_data_handling.ipynb', 'repo_name': 'NijatZeynalov/handling-imbalanced-data', 'size': '14616'}