<jupyter_start><jupyter_text>Credit Card Fraud Detection ![image.png](https://mercobank.com/assets/frontend/img/pages/fp.png) ***Author:*** *<NAME> ****Twitter:*** *[@chesar_che](https://twitter.com/chesar_che)* ***Metadata:*** - This dataset has 122 variables wich simulate a real business scenario such as IncomeTotal, AMTAPPLICATION, AMT_CREDIT, etc.- The information about variables is attached with csv file.- Data is available here: [Credit Card Fraud Detection](https://www.kaggle.com/mishra5001/credit-card?select=application_data.csv) ***Purpose:***- Understanding of risk analytics in banking and financial services and understand how data is used to minimise the risk of losing money while lending to customers. ***Attribute Information*** - Description of attributes is avilable here: [description](https://www.kaggle.com/mishra5001/credit-card?select=columns_description.csv)  💾 Libraries<jupyter_code># pip install altair==4.2.0rc1
# pip install openpyxl
# pip install xgboost
# pip install lightgbm
import pandas as pd
import numpy as np
import datetime as dt
import random

import altair as alt
import seaborn as sns
import matplotlib as mpl
import matplotlib.pyplot as plt

import sklearn as skl
import warnings
warnings.filterwarnings("ignore")
random.seed(14)
print(' ..:: VERSIONS ::..')
print(f' Numpy version: {np.__version__}')
print(f' Pandas version: {pd.__version__}')
print(f' Altair version: {alt.__version__}')
print(f' Seaborn version: {sns.__version__}')
print(f' Matplotlib version: {mpl.__version__}')
print(f' Scikit Learn version: {skl.__version__}')<jupyter_output>..:: VERSIONS ::..
 Numpy version: 1.22.1
 Pandas version: 1.3.4
 Altair version: 4.2.0rc1
 Seaborn version: 0.11.2
 Matplotlib version: 3.5.0
 Scikit Learn version: 1.0.1<jupyter_text>📗 Data Reading<jupyter_code>%%time
data = pd.read_csv('data/data.csv', delimiter=',')
data.head()
data.shape<jupyter_output><empty_output><jupyter_text>- I have 307511 rows and 122 columns in this dataset.<jupyter_code>data.memory_usage(deep=True)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis (EDA)  🔍 Data Inspection<jupyter_code>data.info()
pp = list(np.arange(0.1, 0.9, 0.1)) + list(np.arange(0.9, 1.0, 0.01))
descriptives = data.describe(percentiles = pp)
descriptives<jupyter_output><empty_output><jupyter_text>- There is a suspicion of a outliers presence because the ratio between the last computed percentiles.<jupyter_code>data.isnull().sum().any()<jupyter_output><empty_output><jupyter_text>- There are missing values in the dataset.  🔧 Missing Values Treatment<jupyter_code>missings_df = pd.DataFrame(data.isnull().mean(), columns=['p_missing'])
missings_df.loc[missings_df['p_missing']>0]<jupyter_output><empty_output><jupyter_text>- There are 67 variables with missing values. So, I will remove variables with 15% or more of missing values because they are not recommended for imputation and just obstruct the analysis.<jupyter_code>mask = missings_df['p_missing']>0.15
columns = list(missings_df.loc[mask].index)
def ptg_missing(df, c):
    return (df[str(c)].isnull().mean() * 100)
data_trt_missing = data.copy()
%%time
index = 0
print(' ---------------------------------------------------------------------')
for column in columns:
    if ptg_missing(data_trt_missing, column) > 15:
        print(f'Removing <{column}> column... with <{round(ptg_missing(data_trt_missing, column), 2)}%> of missing values')
        data_trt_missing.drop(str(column), axis=1, inplace=True)
        index+=1
print(' ---------------------------------------------------------------------')
print(f' {index} columns have been removed.')<jupyter_output>---------------------------------------------------------------------
Removing <OWN_CAR_AGE> column... with <65.99%> of missing values
Removing <OCCUPATION_TYPE> column... with <31.35%> of missing values
Removing <EXT_SOURCE_1> column... with <56.38%> of missing values
Removing <EXT_SOURCE_3> column... with <19.83%> of missing values
Removing <APARTMENTS_AVG> column... with <50.75%> of missing values
Removing <BASEMENTAREA_AVG> column... with <58.52%> of missing values
Removing <YEARS_BEGINEXPLUATATION_AVG> column... with <48.78%> of missing values
Removing <YEARS_BUILD_AVG> column... with <66.5%> of missing values
Removing <COMMONAREA_AVG> column... with <69.87%> of missing values
Removing <ELEVATORS_AVG> column... with <53.3%> of missing values
Removing <ENTRANCES_AVG> column... with <50.35%> of missing values
Removing <FLOORSMAX_AVG> column... with <49.76%> of missing values
Removing <FLOORSMIN_AVG> column... with <67.85%> of missing values
Removing <LANDAREA_AVG> column... with <5[...]<jupyter_text>- There were many columns with high percentage of missing values (51).<jupyter_code>data.shape , data_trt_missing.shape<jupyter_output><empty_output><jupyter_text>- Now, I only have these variables:<jupyter_code>data_trt_missing.columns
sns.set()
fig, ax = plt.subplots()
fig.set_size_inches(10,6)
ax.set_title('Distribution of missing values for each row', weight='bold')
sns.distplot(data.isnull().sum(axis =1), color='r')
plt.show()<jupyter_output><empty_output><jupyter_text>- Due to there were many columns with high percentage of missing values, each row had too many missing values.<jupyter_code>sns.set()
fig, ax = plt.subplots()
fig.set_size_inches(10,6)
ax.set_title('Distribution of missing values for each row', weight='bold')
sns.distplot(data_trt_missing.isnull().sum(axis =1), color='g')
plt.show()<jupyter_output><empty_output><jupyter_text>- Before removing columns with high amount of missing values a great part of rows had 50 NaN values and after just a few rows have around 6 NaN values.<jupyter_code>summary = pd.DataFrame()

summary['pct_missings'] = data_trt_missing.isnull().sum() / data_trt_missing.shape[0]
summary['pct_zeros'] = (data_trt_missing == 0).sum() / data_trt_missing.shape[0]
summary['pct_miss_zer'] = summary['pct_missings'] + summary['pct_zeros']

summary.T<jupyter_output><empty_output><jupyter_text>- It's important verify missing values and zero values, because zeros could mess the analysis.<jupyter_code>data_trt_missing.sample(3)
categorical_columns = list(data_trt_missing.select_dtypes('object').columns)
numerical_columns   = list(data_trt_missing.select_dtypes(['int','float']).columns)<jupyter_output><empty_output><jupyter_text>- Categorical columns are correct. There are not filtered continous variables.<jupyter_code>data_trt_missing[numerical_columns].isnull().mean()
data_trt_missing[categorical_columns].isnull().sum()
data_trt_missing['NAME_TYPE_SUITE'].unique()<jupyter_output><empty_output><jupyter_text>- Based on my knowledgment, I can remove NAME_TYPE_SUITE and SK_ID_CURR column because the first is not a confidence variable and the second one is irrelevant.<jupyter_code>len(numerical_columns), len(categorical_columns)
data_trt_missing.drop(['NAME_TYPE_SUITE','SK_ID_CURR'], axis=1, inplace=True)
numerical_columns = list(data_trt_missing.select_dtypes(['int','float']).columns)
categorical_columns = list(data_trt_missing.select_dtypes('object').columns)
len(numerical_columns), len(categorical_columns)
data_trt_missing[(data_trt_missing['AMT_REQ_CREDIT_BUREAU_DAY'].isna())]
data_trt_missing[(data_trt_missing['AMT_REQ_CREDIT_BUREAU_DAY'].isna())]['TARGET'].value_counts()
data_trt_missing.shape
print(f'Missing rows are {round(41519 / 307511 * 100, 2)}% of total.')<jupyter_output>Missing rows are 13.5% of total.<jupyter_text>- I will remove some rows with missing values.- I won't impute them because the computational cost is high.<jupyter_code>data_trt_missing.dropna(inplace=True)
data_trt_missing.shape
sns.set()
fig, ax = plt.subplots()
fig.set_size_inches(3,6)
ax.set_title('Distribution of missing values for each row', weight='bold')
sns.distplot(data_trt_missing.isnull().sum(axis =1), color='b')
plt.show()<jupyter_output><empty_output><jupyter_text>- The dataset doesn't have any missing values. It is ready for analysis or other treatments.  🧰 Outliers Treatment - I will create a dataframe with last percentiles for look possible outliers values:<jupyter_code>data_describe = data_trt_missing.describe(percentiles=pp)
data_describe<jupyter_output><empty_output><jupyter_text>- In this function I exclude variables with values between 0 and 1:<jupyter_code>outliers_max = []

def get_max_outliers_ratio(df):
    df_columns = list(df.columns)
    for column in df_columns:
        max_value = float(df.loc['max', str(column)])
        pp_value = float(df.loc['95%', str(column)])
        if max_value > 1.0:
            outliers_max.append(str(column))
            pp_value = float(df.loc['95%', str(column)])
            if pp_value!=0:
                ratio = max_value / pp_value
            else:
                ratio = -1
            print(f'Ratio of {column} column: {ratio}')
get_max_outliers_ratio(data_describe)
data_trt_missing[outliers_max].describe(percentiles=list(np.arange(0.9,1,0.01))).T<jupyter_output><empty_output><jupyter_text>- I'm going to replace max outliers over the 99° percentile with that percentile value.<jupyter_code>data_trt_outliers = data_trt_missing.copy()
for column in outliers_max:
    data_trt_outliers[str(column)] = np.where(data_trt_outliers[column] > data_describe[column]['99%'], data_describe[column]['99%'], data_trt_outliers[column])
data_trt_outliers[outliers_max].describe(percentiles=list(np.arange(0.9,1,0.01))).T<jupyter_output><empty_output><jupyter_text>- Now, I will check the outliers in minimum values.<jupyter_code>data_describe_min = data_trt_outliers.describe(percentiles=list( np.arange(0.01,0.1,0.01) ))
outliers_min = []

def get_min_outliers_ratio(df):
    df_columns = list(df.columns)
    for column in df_columns:
        max_value = float(df.loc['max', str(column)])
        min_value = float(df.loc['min', str(column)])
        pp_value_01 = float(df.loc['1%', str(column)])
        pp_value_02 = float(df.loc['2%', str(column)])
        pp_value_03 = float(df.loc['3%', str(column)])
        if max_value > 1.0:
            outliers_min.append(str(column))
            if pp_value_01!=0 and pp_value_02!=0 and pp_value_03!=0:
                ratio_01 = min_value / pp_value_01
                ratio_02 = min_value / pp_value_02
                ratio_03 = min_value / pp_value_03
                print(f'Ratios of {column} column: (1% → {ratio_01}) (2% → {ratio_02}) (3% → {ratio_03})')
            else:
                print(f'Min:{min_value} (1% → {pp_value_01}) (2% → {pp_value_02}) (3% → {pp_value_03})')
            
get_min_outliers_ratio(data_describe_min)<jupyter_output>Min:0.0 (1% → 0.0) (2% → 0.0) (3% → 0.0)
Ratios of AMT_INCOME_TOTAL column: (1% → 0.58) (2% → 0.48333333333333334) (3% → 0.38666666666666666)
Ratios of AMT_CREDIT column: (1% → 0.5889281507656066) (2% → 0.4416961130742049) (3% → 0.37533310813346843)
Ratios of AMT_ANNUITY column: (1% → 0.25995655322230266) (2% → 0.23525557011795545) (3% → 0.20467502850627137)
Ratios of AMT_GOODS_PRICE column: (1% → 0.6) (2% → 0.45) (3% → 0.391304347826087)
Ratios of DAYS_EMPLOYED column: (1% → 1.6261063051563873) (2% → 1.9168412784363915) (3% → 2.17378640776699)
Ratios of CNT_FAM_MEMBERS column: (1% → 1.0) (2% → 1.0) (3% → 1.0)
Ratios of REGION_RATING_CLIENT column: (1% → 1.0) (2% → 1.0) (3% → 1.0)
Ratios of REGION_RATING_CLIENT_W_CITY column: (1% → 1.0) (2% → 1.0) (3% → 1.0)
Ratios of HOUR_APPR_PROCESS_START column: (1% → 0.0) (2% → 0.0) (3% → 0.0)
Min:0.0 (1% → 0.0) (2% → 0.0) (3% → 0.0)
Min:0.0 (1% → 0.0) (2% → 0.0) (3% → 0.0)
Min:0.0 (1% → 0.0) (2% → 0.0) (3% → 0.0)
Min:0.0 (1% → 0.0) (2% → 0.0) (3%[...]<jupyter_text>- The succession between values is correct. So, they don't need a imputation.  🧽 Saving cleaned dataframe<jupyter_code>data_cleaned = data_trt_outliers.copy()
data_cleaned.shape
#data_cleaned.to_excel("output/data_treated.xlsx")<jupyter_output><empty_output><jupyter_text>- The cleaned dataset has 39 numerical and 10 categorical columns.  🔖 Data Understanding<jupyter_code>data_cleaned.columns
data_cleaned.sample(4)<jupyter_output><empty_output><jupyter_text>❓ *Which is the most common contract type in fraud cases?*<jupyter_code>data_cleaned.groupby(by=['TARGET','NAME_CONTRACT_TYPE'])[['NAME_CONTRACT_TYPE']].count()<jupyter_output><empty_output><jupyter_text>- The most frequent contract type in fraud cases is Cash loans.  ❓ *Which is the most frequent gender in fraud cases?*<jupyter_code>data_cleaned.groupby(by=['TARGET','CODE_GENDER'])[['CODE_GENDER']].count()<jupyter_output><empty_output><jupyter_text>- In general, women are more frequently than men. But the proportion between men and women in fraud and no fraud cases are different.- In no fraud cases, the proportion is 0.33% men and 0.67% women. While, in fraud cases, the proportion is 42% men and 58% women.  ❓ *Is the number of children related to fraud cases?*<jupyter_code>fig, ax = plt.subplots()

ax = sns.scatterplot(x="TARGET", y="CNT_CHILDREN", data=data_cleaned)
plt.show()<jupyter_output><empty_output><jupyter_text>- The number of children is not related to fraud cases.  ❓ *Which is the most frequent education level in fraud cases?*<jupyter_code>data_cleaned.groupby(by=['TARGET','NAME_EDUCATION_TYPE'])[['NAME_EDUCATION_TYPE']].count()<jupyter_output><empty_output><jupyter_text>- It's very unlikely that a person with an academic degree commit fraud.- The most common education level in fraud cases are Secondary and Higher education.  ❓ *What propierties or documents are frequent in fraud cases?*<jupyter_code>flag_columns = []
data_columns = list(data_cleaned.columns)

for c in data_columns:
    if (c).startswith('FLAG'):
        flag_columns.append(c)
flag_columns
for c in flag_columns:
    display( data_cleaned.groupby(by=c)[['TARGET']].count() )
data_cleaned.FLAG_MOBIL.value_counts(dropna=False)
data_cleaned.FLAG_DOCUMENT_2.value_counts(dropna=False)
data_cleaned.drop(['FLAG_MOBIL','FLAG_DOCUMENT_2'],axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>- It's common that in fraud cases people don't file to have a car but they file to have own realty.  💰  *How is the annual income of clients?*<jupyter_code>fig, ax = plt.subplots()
fig.set_size_inches(14,6)
ax.set_title('Distribution of annual incomes', weight='bold')
ax = sns.distplot(data_cleaned.AMT_INCOME_TOTAL, color='y')
plt.show()<jupyter_output><empty_output><jupyter_text>💰  *How is distribuited the loans amounts?*<jupyter_code>fig, ax = plt.subplots()
fig.set_size_inches(14,6)
ax.set_title('Distribution of loan amounts', weight='bold')
ax = sns.distplot(data_cleaned.AMT_CREDIT, color='g')
plt.show()<jupyter_output><empty_output><jupyter_text>- Both distributions are asymmetrical. So, in the next section I will some especial transformations for this type of variables.  🔃   *How correlated are variables?*<jupyter_code>sns.heatmap(data_cleaned.corr())
plt.show()<jupyter_output><empty_output><jupyter_text>🛠️ Feature Engineering<jupyter_code>data_cleaned.shape<jupyter_output><empty_output><jupyter_text>- Divide categorical and numeric columns:<jupyter_code>categorical_columns = list(data_cleaned.select_dtypes('object').columns)
numerical_columns   = list(data_cleaned.select_dtypes(['int','float']).columns)<jupyter_output><empty_output><jupyter_text>- I will create new features:<jupyter_code>data_cleaned['PCTG_ANNUITY'] = data_cleaned['AMT_ANNUITY'] / data_cleaned['AMT_CREDIT']
data_cleaned['PCTG_GOODS'] = data_cleaned['AMT_GOODS_PRICE'] / data_cleaned['AMT_INCOME_TOTAL']
data_cleaned['INCOME_CHILD'] = data_cleaned['AMT_INCOME_TOTAL'] / (data_cleaned['CNT_CHILDREN'] + 1)

data_cleaned['POSSESSIONS'] = data_cleaned.apply(lambda x: round( (x.FLAG_EMP_PHONE + x.FLAG_WORK_PHONE + x.FLAG_CONT_MOBILE + x.FLAG_PHONE + x.FLAG_EMAIL) ,2),axis=1)
data_cleaned.shape
categorical_columns = list(data_cleaned.select_dtypes('object').columns)
numerical_columns   = list(data_cleaned.select_dtypes(['int','float']).columns)
len(numerical_columns)<jupyter_output><empty_output><jupyter_text>- I will check assymetrical columns and how assymetrical they are:<jupyter_code>column_skewness = []
for c in numerical_columns:
    skewness = data_cleaned[str(c)].skew()
    print(f'Column <<{str(c)}>> → Skewness <{skewness}>')
    if abs(skewness) > 1:
        column_skewness.append(str(c))
len(column_skewness)<jupyter_output><empty_output><jupyter_text>- I will transform some columns which are relevant for the analysis.- I will use boxcox transformation because those columns have positive values greater than 0.<jupyter_code>cols_trans = ['AMT_INCOME_TOTAL','AMT_CREDIT','AMT_GOODS_PRICE','PCTG_ANNUITY','INCOME_CHILD','PCTG_GOODS']
from scipy.stats import boxcox

for c in cols_trans:
    bc_trans = boxcox(data_cleaned[str(c)])[0]
    data_cleaned['BC_' + c] = bc_trans
data_cleaned_numeric = data_cleaned.copy()
data_cleaned_numeric.shape<jupyter_output><empty_output><jupyter_text>- I will transform categorical columns with a label encoder because if I use an One Hot Encoder, the matrix will be less interpretable and bigger.<jupyter_code>from sklearn.preprocessing import LabelEncoder

for c in categorical_columns:
    print(f'Column: <<{c}>>')
    le = LabelEncoder()
    le.fit(data_cleaned_numeric[str(c)])
    data_cleaned_numeric[str(c)] = le.transform(data_cleaned_numeric[str(c)])
data_cleaned_numeric.shape
len(data_cleaned_numeric.select_dtypes(['int','float']).columns)
data_cleaned_numeric_final = data_cleaned_numeric.copy()<jupyter_output><empty_output><jupyter_text>🕵️ Feature Selection  🌳 *Feature importances with Random Forest Algorithm*<jupyter_code>%%time
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier()
x = data_cleaned_numeric_final.drop('TARGET',axis=1)
y = data_cleaned_numeric_final.TARGET
forest.fit(x, y)
importances = forest.feature_importances_ 
importances_table = pd.concat([pd.DataFrame({'Driver':list(data_cleaned_numeric_final.columns)}),
                              pd.DataFrame({'Importance':list(forest.feature_importances_)})], axis = 1)
importances_variables = importances_table[['Driver','Importance']].sort_values('Importance', ascending = False).reset_index(drop = True)
importances_variables
fig, ax = plt.subplots()
fig.set_size_inches(14,20)
ax.set_title('Feature Importances', weight='bold')
sns.barplot(x="Importance", y="Driver", data=importances_variables, color="g")
plt.show()
subset = list(importances_variables.iloc[0:16, 0].values)
subset.append('TARGET')
subset
df_subset = data_cleaned_numeric_final[subset]<jupyter_output><empty_output><jupyter_text>🖥️ *With Scikit Learn library:*<jupyter_code>%%time
from sklearn.feature_selection import SelectKBest, mutual_info_classif

X = data_cleaned_numeric_final.drop('TARGET', axis=1)
target = data_cleaned_numeric_final.TARGET

bestfeatures = SelectKBest(score_func=mutual_info_classif, k=10)
fit = bestfeatures.fit(X, target,)
dfscores = pd.DataFrame(fit.scores_)
dfcolumns = pd.DataFrame(X.columns) 
featureScores = pd.concat([dfcolumns,dfscores],axis=1)
featureScores.columns = ['Feature','Score']
print(featureScores.nlargest(15,'Score'))
top_subset = ['FLAG_CONT_MOBILE','FLAG_EMP_PHONE','NAME_EDUCATION_TYPE','NAME_HOUSING_TYPE',
             'REGION_RATING_CLIENT_W_CITY','REGION_RATING_CLIENT','FLAG_DOCUMENT_3',
              'FLAG_OWN_REALTY','NAME_FAMILY_STATUS','NAME_INCOME_TYPE','BC_PCTG_ANNUITY',
             'PCTG_ANNUITY','POSSESSIONS','CNT_FAM_MEMBERS','DAYS_EMPLOYED']

top_subset.append('TARGET')
df_top_subset = data_cleaned_numeric_final[top_subset]<jupyter_output><empty_output><jupyter_text>- After I have been trying with features selected by the random forest and the scikit learn library for feature selection, the library achieve a better performance.  📏 Standardization - I reset the index because before I removed some rows with missing values and indexes are not ordered. So, when I try to concatenate both dataframes, pandas will throw an error.- I want to standardize values because all of them have multiple scales, like that the algoritm won't atributte more weight to some of them.<jupyter_code>df_top_subset_index = df_top_subset.reset_index()
predictors = df_top_subset_index.drop(['TARGET','index'], axis=1)
target = df_top_subset_index.TARGET
predictors.head(10)<jupyter_output><empty_output><jupyter_text>- I can use a Standard Scaler because I don't have many missing values.- They were recodified with the 99° percentile.<jupyter_code>from sklearn.preprocessing import StandardScaler

sc = StandardScaler().fit(predictors)

scaled_predictors = sc.transform(predictors)
df_scaled_predictors = pd.DataFrame(scaled_predictors, columns=predictors.columns)
df_scaled_predictors.head(10)
dataset = pd.concat([df_scaled_predictors, target], axis=1)<jupyter_output><empty_output><jupyter_text>✂️ Data Partition<jupyter_code>from sklearn.model_selection import train_test_split

features = dataset.drop('TARGET', axis=1)
y = dataset['TARGET']

X_train, X_test, y_train, y_test = train_test_split(features, 
                                                    y, 
                                                    test_size=0.20,
                                                    stratify = dataset.TARGET,
                                                    random_state=10)
y_train.mean() , y_test.mean()<jupyter_output><empty_output><jupyter_text>***Logistic Regression:***<jupyter_code>from sklearn.linear_model import LogisticRegression

lr = LogisticRegression()
lr.fit(X_train, y_train)
from sklearn.metrics import classification_report, confusion_matrix
fig, ax = plt.subplots()
ax = sns.countplot(x='TARGET', data=dataset)
plt.show()
y_pred_train = lr.predict(X_train)
y_pred_test = lr.predict(X_test)
confusion_matrix(y_train, y_pred_train)
print(classification_report(y_train, y_pred_train))
from sklearn.metrics import roc_auc_score

def evaluate_model(x,y):
    lr = LogisticRegression()
    lr.fit(x, y)
    y_pred = lr.predict(x)
    print('--------------------------------------------------------')
    print(' ..:: ROC AUC SCORE ::..')
    print(f'{roc_auc_score(y, lr.predict_proba(x)[:,1])}')
    print('--------------------------------------------------------')
    print(' ..:: CONFUSSION MATRIX ::..')
    print(f'{confusion_matrix(y, y_pred)}')
    print('--------------------------------------------------------')
    print('\t       ..:: CLASSIFICATION REPORT ::..')
    print(classification_report(y, y_pred))<jupyter_output><empty_output><jupyter_text>- Seemingly, the Logistic Regression is a very simple model for this case. So, I will balance classes a bit (the next algoritm don't require balanced classes) and try with other algorithm (Light GBM).  ⚖️ Class Balancing<jupyter_code>from imblearn.over_sampling import SMOTE
smote = SMOTE(sampling_strategy = 0.10)
X_sm, y_sm = smote.fit_resample(X_train, y_train)
fig, ax = plt.subplots()
data_smote = pd.DataFrame({'TARGET':y_sm})
ax = sns.countplot(x='TARGET', data=data_smote)
plt.show()<jupyter_output><empty_output><jupyter_text>📈 Light GBM<jupyter_code>import lightgbm as lgb
lgb_classifier = lgb.LGBMClassifier(max_depth=3,
                                    num_leaves=6,
                                    objective='binary',
                                    is_unbalance=True).fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>***Train test:***<jupyter_code>y_pred_train_lgb = lgb_classifier.predict(X_train)

print('--------------------------------------------------------')
print(' ..:: ROC AUC SCORE ::..')
print(f'{roc_auc_score(y_pred_train_lgb, lgb_classifier.predict_proba(X_train)[:,1])}')
print('--------------------------------------------------------')
print(' ..:: CONFUSSION MATRIX ::..')
print(f'{confusion_matrix(y_train, y_pred_train_lgb)}')
print('\t       ..:: CLASSIFICATION REPORT ::..')
print(classification_report(y_train, y_pred_train_lgb))
lgb_sm_classifier = lgb.LGBMClassifier(max_depth=3,
                                    num_leaves=6,
                                    objective='binary',
                                    learning_rate=0.1,
                                    is_unbalance=True).fit(X_sm, y_sm)

y_pred_sm_lgb = lgb_sm_classifier.predict(X_sm)

print('--------------------------------------------------------')
print(' ..:: ROC AUC SCORE ::..')
print(f'{roc_auc_score(y_pred_sm_lgb, lgb_classifier.predict_proba(X_sm)[:,1])}')
print('--------------------------------------------------------')
print(' ..:: CONFUSSION MATRIX ::..')
print(f'{confusion_matrix(y_sm, y_pred_sm_lgb)}')
print('\t       ..:: CLASSIFICATION REPORT ::..')
print(classification_report(y_sm, y_pred_sm_lgb))<jupyter_output>--------------------------------------------------------
 ..:: ROC AUC SCORE ::..
0.9890031294517343
--------------------------------------------------------
 ..:: CONFUSSION MATRIX ::..
[[124550  70590]
 [  7065  12449]]
	       ..:: CLASSIFICATION REPORT ::..
              precision    recall  f1-score   support

           0       0.95      0.64      0.76    195140
           1       0.15      0.64      0.24     19514

    accuracy                           0.64    214654
   macro avg       0.55      0.64      0.50    214654
weighted avg       0.87      0.64      0.72    214654<jupyter_text>⛏️ Tuning parameters<jupyter_code>from sklearn.model_selection import GridSearchCV
cv_params = {'max_depth': [2,3,4],
            'n_estimators':[100, 200, 300, 400],
            'learning_rate':[0.01,0.001],
            'feature_fraction':[0.70,0.75,0.80],
            'bagging_fraction':[0.70,0.75,0.80],
            'learning_rate':[0.1,0.01],
             'num_leaves':[6,7,8,9]}
optimized_model = GridSearchCV(lgb_sm_classifier, cv_params, scoring='recall', cv = 4)
%%time  2:30 hours
optimized_model.fit(X_sm, y_sm)
print(optimized_model.best_params_)
print(optimized_model.best_score_) # 0.668343393566961<jupyter_output><empty_output><jupyter_text>✅ Optimized model<jupyter_code>best_lgb = lgb.LGBMClassifier(n_estimators=400,
                              max_depth=4,
                              num_leaves=9,
                              objective='binary',
                              learning_rate=0.1,
                              feature_fraction=0.75,
                              bagging_fraction=0.7,
                              is_unbalance=True,
                              random_state=12)
best_lgb.fit(X_sm, y_sm)
y_pred_lgb_train_best = best_lgb.predict(X_train)
y_pred_sm_lgb_test_best = best_lgb.predict(X_test)
print('--------------------------------------------------------')
print(' ..:: ROC AUC SCORE ::..')
print(f'{roc_auc_score(y_pred_sm_lgb_test_best, best_lgb.predict_proba(X_test)[:,1])}')
print('--------------------------------------------------------')
print(' ..:: CONFUSSION MATRIX ::..')
print(f'{confusion_matrix(y_test, y_pred_sm_lgb_test_best)}')
print('\t       ..:: CLASSIFICATION REPORT ::..')
print(classification_report(y_test, y_pred_sm_lgb_test_best))
fig, ax = plt.subplots()
fig.set_size_inches(5,3)

ax.set_title('Confusion Matrix (TEST)', weight='bold')
conf_mat = confusion_matrix(y_test, y_pred_sm_lgb_test_best)
sns.heatmap(conf_mat, annot=True)
plt.show()<jupyter_output><empty_output><jupyter_text>💽 Saving final model<jupyter_code>import pickle

with open("output/model.pickle", "wb") as wb:
    pickle.dump(best_lgb, wb)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15527065527065528, 'fraction_numerical': 0.030718094820658924, 'mean_word_length': 4.247029513223457, 'pattern_counts': {'":': 0, '<': 220, '<?xml version=': 0, '>': 226, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '47', 'dir': 'jupyter-structured-clean-dedup', 'id': '79394', 'license': 'Apache-2.0', 'n_tokens_mistral': 10055, 'n_tokens_neox': 9395, 'n_words': 1983, 'path': 'Notebook.ipynb', 'repo_name': 'Chesar832/Fraud_Detection_in_Python', 'size': '27390'}