<jupyter_start><jupyter_text>Feature engineeringEncode categorical variables with frequency encoding which is better for decision trees as compared to one-hot encoding. It is difficult for decision trees to look for a good split point where there are very few split points available (such as binary 0/1 in the case of one-hot encoding).To ensure no leaks from the test set, frequencies are obtained only from the training set.Feature crosses to create interaction terms for important features, which are obtained from the model (goodness of a split point, how much impurity is reduced). This helps to model non-linearities in feature space by combining two or more features together.<jupyter_code>def encode(df, col, encoder):
    df[col] = df[col].map(encoder).fillna(0)
    assert not df[col].isna().any()


cols = CATEGORICALS 
for col in tqdm(cols):
    encoder = dict(train[col].value_counts(normalize=True))
    encode(train, col, encoder)
    encode(test, col, encoder)
train[cols].head()
test[cols].head()
def feature_cross(df, left, right, column):
    df[column] = df[left] * df[right]
    
    
for left, right in pairs:
    col = f'{left}_{right}'
    feature_cross(train, left, right, col)
    feature_cross(test, left, right, col)
    FEATURES.add(col)<jupyter_output><empty_output><jupyter_text>Feature Selection[KS Test](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) to test whether two samples are drawn from the distribution. Both samples are drawn from the training set (ensure no data leaks from the test set).Drop features where p value is less than or equals level of significance alpha (10%). These features are dropped because they exhibit too much variance across different samples.<jupyter_code>def _select_features(df1, df2, features, alpha):
    res = []
    for f in tqdm(features):
        if ks_2samp(df1[f], df2[f]).pvalue > alpha:
            res.append(f)
    return res


ALPHA = 0.1
sample = train.sample(frac=1)
mid = int(len(sample) / 2)
original = set(FEATURES)
FEATURES = _select_features(sample[:mid], sample[mid:], FEATURES, ALPHA)
dropped = original - set(FEATURES)
print(f"""
Keep {len(FEATURES)} features={FEATURES}

Drop {len(dropped)} features={dropped}
""")
y_train = train[TARGET]
y_test = test[TARGET]
x_train = train[FEATURES]
x_test = test[FEATURES]<jupyter_output><empty_output><jupyter_text>ModelsTree-based methods; LightGBM is a boosting ensemble designed to reduce bias, whereas Random Forest is a bagging ensemble designed to reduce variance.Advantages of tree-based methods: Cheap to build and make inference (cost grows with tree depth). Non-parametric: does not assume any probability distribution type. Robust to noise and redundant features because outliers are unlikely to be the split point. Unlike linear models and neural nets, we do not need to perform standardisation or attempt to map the features to a normal distribution.Tune the following hyperparameters to avoid overfitting: minimum number of examples in leaf, number of features to consider for splitting, number of examples available to grow each tree.<jupyter_code>%%time
# scale_pos_weight is the ratio of number of negative class to the positive class.
model = lgb.LGBMClassifier(n_estimators=ESTIMATORS, reg_alpha=1, scale_pos_weight=49)
pipe = Pipeline([('model', model)])
param_grid = {
    'model__learning_rate': [0.01, 0.001],
    'model__subsample': [0.01, 0.05, 0.1],
    'model__min_child_samples': [20, 40, 80],
    'model__colsample_bytree': [0.01, 0.05, 0.1]
}
# Defaults to StratifiedKFold
lgbm = GridSearchCV(pipe, cv=FOLDS, param_grid=param_grid, scoring=SCORING)
lgbm.fit(x_train, y_train)
best_score = lgbm.best_score_
best_score_std = lgbm.cv_results_['std_test_score'][lgbm.best_index_]
print(f"""LGBM
best params={repr(lgbm.best_params_)}
best score={best_score:0.3f}, std={best_score_std:0.3f}
""")
res = pd.DataFrame.from_dict(lgbm.cv_results_)
res.set_index('rank_test_score', inplace=True)
res.sort_index(inplace=True)
res.head(100)
lgb.plot_importance(lgbm.best_estimator_['model'], figsize=(10, 80))
%%time
model = RandomForestClassifier(n_estimators=ESTIMATORS, class_weight='balanced_subsample')
pipe = Pipeline([('model', model)])
param_grid = {
    'model__min_samples_leaf': [0.01, 0.02, 0.04],
    'model__max_features': ['log2'],
    'model__max_samples': [0.01, 0.05, 0.1],
}
# Defaults to StratifiedKFold
rf = GridSearchCV(pipe, cv=FOLDS, param_grid=param_grid, scoring=SCORING)
rf.fit(x_train, y_train)
best_score = rf.best_score_
best_score_std = rf.cv_results_['std_test_score'][rf.best_index_]
print(f"""Random Forest
best params={repr(rf.best_params_)}
best score={best_score:0.3f}, std={best_score_std:0.3f}
""")
res = pd.DataFrame.from_dict(rf.cv_results_)
res.set_index('rank_test_score', inplace=True)
res.sort_index(inplace=True)
res.head(100)
importances = rf.best_estimator_['model'].feature_importances_
plt.figure(figsize=(10,80))
plt.title("Feature importances")
indices = np.argsort(importances)

plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), [FEATURES[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()<jupyter_output><empty_output><jupyter_text>EvaluationStratified 5-fold cross validation is important because we have a class imbalance problem (positive class is present at 2% of training set, 1% of test set). K-folds allows all examples to be used for validation once and for training (k - 1) times. We reduce the risk of the result having a heavy dependency on the membership of the training and validation sets.Validation- LGBM best score=0.710, std=0.087- Random Forest best score=0.715, std=0.072(Important to indicate the standard deviation so that we can assess how the models will fare on unseen data)Based on the validation results, RF seems to be the better model because it has a slightly higher score and smaller standard deviation. The test set shows that the LGBM Model is able to generalize slightly better than random forest.auc_lgbm=0.635, auc_rf=0.615<jupyter_code>test['y_lgbm'] = lgbm.predict_proba(x_test)[:,1]
test['y_rf'] = rf.predict_proba(x_test)[:,1]
test['y_blend'] = 0.7 * test['y_lgbm'] + 0.3 * test['y_rf']
auc_lgbm = roc_auc_score(y_test, test['y_lgbm'], average='macro')
auc_rf = roc_auc_score(y_test, test['y_rf'], average='macro')
auc_blend = roc_auc_score(y_test, test['y_blend'], average='macro')
print(f'auc_blend={auc_blend:0.3f}, auc_lgbm={auc_lgbm:0.3f}, auc_rf={auc_rf:0.3f}')
res = test[test[TARGET] == 1].copy()
res.sort_values(by=['y_lgbm'], inplace=True, ascending=False)
res[[TARGET, 'y_blend', 'y_lgbm', 'y_rf']].head(10)
sub = test[[TARGET, 'y_blend', 'y_lgbm', 'y_rf']]
sub.info()
sub.head()
sub.to_csv('submission.csv', index=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11864655046140325, 'fraction_numerical': 0.019627947854108686, 'mean_word_length': 6.061013443640124, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '362606', 'license': 'MIT', 'n_tokens_mistral': 2359, 'n_tokens_neox': 2260, 'n_words': 664, 'path': 'ex2.ipynb', 'repo_name': 'seahrh/credit-scoring-exercise', 'size': '6827'}