<jupyter_start><jupyter_text>Split columns into quantitative and category<jupyter_code>all_columns = set(train_df.columns) | set(test_df.columns)
all_quan_cols = set(train_df.describe().columns) | set(test_df.describe().columns)

all_cat_cols = all_columns - all_quan_cols
all_cat_cols
train_df = pd.get_dummies(train_df, columns=all_cat_cols)
test_df = pd.get_dummies(test_df, columns=all_cat_cols)

check_columns()<jupyter_output>columns not match ERROR<jupyter_text>Fix missing for the debt_settlement_flag column<jupyter_code>test_df["debt_settlement_flag_Y"] = 0

check_columns()<jupyter_output>columns match<jupyter_text>Train model to be detect if the loans are high risk<jupyter_code>target = "loan_status_high_risk"
# split train data
x_train  = train_df.drop(columns=[target])
y_train = train_df[[target]].values.ravel()

print(x_train.shape, y_train.shape)
# split test data
x_test  = test_df.drop(columns=[target])
y_test = test_df[[target]].values.ravel()
#
print(x_test.shape, y_test.shape)<jupyter_output>(4702, 95) (4702,)<jupyter_text>Make a prediction as to which model you think will perform better Random Forest will perform better as risk prediction may not linearly depend on the explantory variables. With scaling, logistic regression may perform better.<jupyter_code># score for log reg
logReg = LogisticRegression(
    solver='lbfgs', 
    max_iter=50,
    random_state=0
)
logReg.fit(x_train, y_train)
print("LogisticRegression score: ", logReg.score(x_test, y_test))
# score for random forest classifier
randForClass = RandomForestClassifier(random_state=0)
randForClass.fit(x_train, y_train)
print("RandomForestClassifier score: ", randForClass.score(x_test, y_test))<jupyter_output>RandomForestClassifier score:  1.0<jupyter_text>Try models again with scaling<jupyter_code>scaler = StandardScaler()
scaler.fit(x_train)

x_train_scaled = scaler.transform(x_train)
x_test_scaled = scaler.transform(x_test)
# score for log reg
logReg = LogisticRegression(
    solver='lbfgs', 
    max_iter=50,
    random_state=0
)
logReg.fit(x_train_scaled, y_train)
print("LogisticRegression score: ", logReg.score(x_test_scaled, y_test))
# score for random forest classifier
randForClass = RandomForestClassifier(random_state=0)
randForClass.fit(x_train_scaled, y_train)
print("RandomForestClassifier score: ", randForClass.score(x_test_scaled, y_test))<jupyter_output>RandomForestClassifier score:  1.0
{'fraction_non_alphanumeric': 0.09937888198757763, 'fraction_numerical': 0.009523809523809525, 'mean_word_length': 7.948148148148148, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '482677', 'license': 'MIT', 'n_tokens_mistral': 881, 'n_tokens_neox': 851, 'n_words': 168, 'path': 'supervised-learning_predicting-credit-risk.ipynb', 'repo_name': 'jameswniu/Predicting-Credit-Risk', 'size': '2415'}