<jupyter_start><jupyter_text>Uploading, Cleaning Data<jupyter_code>df = pd.read_csv('application_data_min.csv')
# Cleaning for features that are used in dataset
df['CNT_FAM_MEMBERS'].describe()
df.loc[df['CNT_FAM_MEMBERS'].isnull()] = 0
df = df.dropna(subset = ['AMT_ANNUITY'])<jupyter_output><empty_output><jupyter_text>Target Analysis<jupyter_code># TARGET = our predicted value, 1 = default, 0 = no default
y = df['TARGET']
print(y.mean())
print(y.value_counts())<jupyter_output>0.08073196986006459
0    282674
1     24825
Name: TARGET, dtype: int64<jupyter_text>Preprocessing Data For our classification model, we chose the following features:- AMT_INCOME_TOTAL- AMT_CREDIT- AMT_ANNUITY- CODE_GENDER- CNT_FAM_MEMBERS- NAME_CONTRACT_TYPE- NAME_EDUCATION_TYPEWe wanted to limit the number of features that the user would have to input in our application. In addition, we wanted to choose features that we believed could be relevant in predicting card defaults/delayed card payments, as well as features that could be easily remembered by users.<jupyter_code># Columns: Count_fam_members, income, credit, annuity, contract type, gender, education
X = df.drop(columns = ['TARGET'])
X = pd.get_dummies(X, drop_first = True)<jupyter_output><empty_output><jupyter_text>Classification Choices Logistic Regression, Gradient Boosted Trees?, Decision Tree Classifier, KNN<jupyter_code>classification_models = [LogisticRegression(), KNeighborsClassifier(), DecisionTreeClassifier(), RandomForestClassifier()]<jupyter_output><empty_output><jupyter_text>Classification w/ Class Imbalance First, we attempted to fit an SGD Classifier to the dataset. We also fit a logistic regression model to our data. However, because only ~8% of the data defaulted, we saw an obvious class imbalance. This led to our model predicting very few defaults (if at all).<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, random_state = 42)
MLA_columns = []
MLA_compare = pd.DataFrame(columns = MLA_columns)
row_index = 0

for alg in classification_models:
    predicted = alg.fit(X_train, y_train).predict(X_test)
    MLA_name = alg.__class__.__name__
    MLA_compare.loc[row_index,'Model Name'] = MLA_name
    MLA_compare.loc[row_index, 'Train Accuracy'] = round(alg.score(X_train, y_train), 2)
    MLA_compare.loc[row_index, 'Test Accuracy'] = round(alg.score(X_test, y_test), 2)
    MLA_compare.loc[row_index, 'Precision'] = round(precision_score(y_test, predicted),2)
    MLA_compare.loc[row_index, 'Recall'] = round(recall_score(y_test, predicted),2)
    MLA_compare.loc[row_index, 'F1 score'] = round(f1_score(y_test, predicted),2)
    row_index+=1
    MLA_compare.sort_values(by = ['Test Accuracy'], ascending = False, inplace = True)  
    
MLA_compare<jupyter_output>/opt/anaconda3/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))<jupyter_text>From this, best one is KNeighborsClassifier - close train and test accuracy so probably no overfitting, precision and recall not equal to 0<jupyter_code>KNN = KNeighborsClassifier()
KNN.fit(X_train, y_train)

train_pred = KNN.predict(X_train)
test_pred = KNN.predict(X_test)
confusion_matrix(y_train, train_pred)
confusion_matrix(y_test, test_pred)
cross_val_score(KNN, X, y, cv = 5)<jupyter_output><empty_output><jupyter_text>Classification w/ Random Oversampling<jupyter_code>oversample = RandomOverSampler(sampling_strategy='minority', random_state = 42)
X_train_o, y_train_o = oversample.fit_resample(X_train, y_train)
print('After Oversampling:', Counter(y_train_o))
MLA_columns = []
MLA_compare = pd.DataFrame(columns = MLA_columns)
row_index = 0

for alg in classification_models:
    predicted = alg.fit(X_train_o, y_train_o).predict(X_test)
    MLA_name = alg.__class__.__name__
    MLA_compare.loc[row_index,'Model Name'] = MLA_name
    MLA_compare.loc[row_index, 'Train Accuracy'] = round(alg.score(X_train_o, y_train_o), 2)
    MLA_compare.loc[row_index, 'Test Accuracy'] = round(alg.score(X_test, y_test), 2)
    MLA_compare.loc[row_index, 'Precision'] = round(precision_score(y_test, predicted),2)
    MLA_compare.loc[row_index, 'Recall'] = round(recall_score(y_test, predicted),2)
    MLA_compare.loc[row_index, 'F1 score'] = round(f1_score(y_test, predicted),2)
    row_index+=1
    MLA_compare.sort_values(by = ['Test Accuracy'], ascending = False, inplace = True)  
    
MLA_compare<jupyter_output><empty_output><jupyter_text>From this, we think that DecisionTreeClassifier is the best - while test accuracy is lower, it has a higher recall and f1 score.<jupyter_code>dt_o = DecisionTreeClassifier().fit(X_train_o, y_train_o)
predict_train = dt_o.predict(X_train_o)

dt_o.score(X_train_o, y_train_o)
confusion_matrix(predict_train, y_train_o)
predict_test = dt_o.predict(X_test)
dt_o.score(X_test, y_test)
confusion_matrix(predict_test, y_test)
cross_val_score(dt_o, X, y, cv = 5)<jupyter_output><empty_output><jupyter_text>Classification with SMOTE<jupyter_code>smote = SMOTE(sampling_strategy='minority', random_state = 42)
X_train_s, y_train_s = smote.fit_resample(X_train, y_train)
MLA_columns = []
MLA_compare = pd.DataFrame(columns = MLA_columns)
row_index = 0

for alg in classification_models:
    predicted = alg.fit(X_train_s, y_train_s).predict(X_test)
    MLA_name = alg.__class__.__name__
    MLA_compare.loc[row_index,'Model Name'] = MLA_name
    MLA_compare.loc[row_index, 'Train Accuracy'] = round(alg.score(X_train_s, y_train_s), 2)
    MLA_compare.loc[row_index, 'Test Accuracy'] = round(alg.score(X_test, y_test), 2)
    MLA_compare.loc[row_index, 'Precision'] = round(precision_score(y_test, predicted),2)
    MLA_compare.loc[row_index, 'Recall'] = round(recall_score(y_test, predicted),2)
    MLA_compare.loc[row_index, 'F1 score'] = round(f1_score(y_test, predicted),2)
    row_index+=1
    MLA_compare.sort_values(by = ['Test Accuracy'], ascending = False, inplace = True)  
    
MLA_compare<jupyter_output><empty_output><jupyter_text>Decision tree classifier is the best? - second highest test accuracy, but better in terms of recall and f1 score than RandomForestClassifier<jupyter_code>dt_s = DecisionTreeClassifier().fit(X_train_s, y_train_s)
predict_train = dt_s.predict(X_train_s)

dt_s.score(X_train_s, y_train_s)
confusion_matrix(predict_train, y_train_s)
predict_test = dt_s.predict(X_test)
dt_s.score(X_test, y_test)
confusion_matrix(y_test, test_pred)
cross_val_score(dt_s, X, y, cv = 5)<jupyter_output><empty_output><jupyter_text>Classfication with Undersampling Note: results are going to change drastically once they're sampled<jupyter_code>X_y_train = X_y_train = pd.concat([X_train.reset_index(drop = True),
                       y_train.reset_index(drop = True)], axis = 1)

# Create data sets for defaults and non-defaults
nondefaults = X_y_train[X_y_train['TARGET'] == 0]
defaults = X_y_train[X_y_train['TARGET'] == 1]

# Undersample the non-defaults
nondefaults_under = nondefaults.sample(len(defaults), random_state = 42)

# Concatenate the undersampled nondefaults with defaults
X_y_train_undersampled = pd.concat([nondefaults_under.reset_index(drop = True),
                             defaults.reset_index(drop = True)], axis = 0)
X_train_u = X_y_train_undersampled.drop(columns = ['TARGET'])
y_train_u = X_y_train_undersampled['TARGET']
MLA_columns = []
MLA_compare = pd.DataFrame(columns = MLA_columns)
row_index = 0

for alg in classification_models:
    predicted = alg.fit(X_train_u, y_train_u).predict(X_test)
    MLA_name = alg.__class__.__name__
    MLA_compare.loc[row_index,'Model Name'] = MLA_name
    MLA_compare.loc[row_index, 'Train Accuracy'] = round(alg.score(X_train_s, y_train_s), 2)
    MLA_compare.loc[row_index, 'Test Accuracy'] = round(alg.score(X_test, y_test), 2)
    MLA_compare.loc[row_index, 'Precision'] = round(precision_score(y_test, predicted),2)
    MLA_compare.loc[row_index, 'Recall'] = round(recall_score(y_test, predicted),2)
    MLA_compare.loc[row_index, 'F1 score'] = round(f1_score(y_test, predicted),2)
    row_index+=1
    MLA_compare.sort_values(by = ['Test Accuracy'], ascending = False, inplace = True)  
    
MLA_compare<jupyter_output><empty_output><jupyter_text>For this one, RandomForestClassifier is the best - best test accuracy, precision, recall, and f1 score<jupyter_code>rfc_u = RandomForestClassifier()
rfc_u.fit(X_train_u, y_train_u)

train_pred = rfc_u.predict(X_train_u)
train_all_pred = rfc_u.predict(X_train)
test_pred = rfc_u.predict(X_test)
confusion_matrix(y_train_u, train_pred)
confusion_matrix(y_train, train_all_pred)
confusion_matrix(y_test, test_pred)
cross_val_score(rfc_u, X, y, cv = 5)<jupyter_output><empty_output><jupyter_text>Summary Overall, best seems to be the Decision Tree classifier with random oversampling - it has a relatively high test accuracy (0.82), recall of 0.1, and precision of 0.17.<jupyter_code>dt_o = DecisionTreeClassifier().fit(X_train_o, y_train_o)
predict_train = dt_o.predict(X_train_o)

dt_o.score(X_train_o, y_train_o)
dt_o.feature_importances_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10778897194236811, 'fraction_numerical': 0.01191636875744773, 'mean_word_length': 6.548650858544563, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '549439', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3459, 'n_tokens_neox': 3318, 'n_words': 689, 'path': 'Application Data Classification.ipynb', 'repo_name': 'CMU-IDS-2020/a3-nathan-juliette-a3', 'size': '9231'}