<jupyter_start><jupyter_text>Extra TreesIn this file, Extra Trees model is fit by tuning different hyperparameters based on F_{1.5} score: Criterion for splitting, number of trees, minimum number of samples to split, and maximum number of features for choosing to split.We will look at the *slopes* of precision and recall across different hyperparameters to see its influence rather than their face-values since the cut-off probability is set to 0.5 for the tuning process before deciding the threshold probability.Also, by comparing the fit to the training set vs. validation set, it is easy to see how trees tend to overfit. Lastly, the cut-off probability is decided by the F score. Import the modules<jupyter_code>from sklearn.ensemble import ExtraTreesClassifier
from ipynb.fs.full.helper_functions import * # Custom function to plot the results
import matplotlib.pyplot as plt

pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)<jupyter_output><empty_output><jupyter_text>Import the imputed dataset<jupyter_code>X_train, X_val, y_train, y_val = pd.read_pickle('data/imp_train.pkl'), pd.read_pickle('data/imp_val.pkl'), pd.read_pickle('data/y_train.pkl'), pd.read_pickle('data/y_val.pkl')<jupyter_output><empty_output><jupyter_text>Hyperparameter tuning1. Criterion = {“gini”, “entropy”}2. min_samples_split from 2 to 25(the minimum number of samples required to split an internal node)3. n_estimators from 5 to 100(the number of trees)4. max_features from 8 to 40(the maximum number of features to choose from when splitting)- Remark: Just like Random Forest, both maximum number of trees and number of trees plateaus after a small number and only minimum-samples-to-split has a negative effect to the precision. It seems that tree algorithms are sensitive to minimum-samples-to-split.  1. Criterion Criterion = ***Gini***with 10 estimators, 20 min_samples_split, sqrt( of features) max_features- F score is .96 in the training set, but .58 in the evaluation set. This is strongly overfit.<jupyter_code>extra_trees = ExtraTreesClassifier(random_state = 10,
                                   class_weight = {0: 1, 1: 27},
                                   bootstrap = False,
                                   criterion = 'gini',
                                   n_estimators = 10,
                                   min_samples_split = 20,
                                   max_features = "sqrt")

extra_trees.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Fit to the training set<jupyter_code># Fit to the training set
y_probs = extra_trees.predict_proba(X_train)
precs, recs, fbetas, acc_scores = plot_fbeta_recall_precision_acc(y_probs[:,1], y_train)<jupyter_output>Best Fbeta(1.5) score  0.96236 at prob decision threshold >=  0.68973<jupyter_text>Fit to the validation set<jupyter_code># Fit to the validation set
y_probs = extra_trees.predict_proba(X_val)
precs, recs, fbetas, acc_scores = plot_fbeta_recall_precision_acc(y_probs[:,1], y_val)<jupyter_output>Best Fbeta(1.5) score  0.58408 at prob decision threshold >=  0.54604<jupyter_text>Criterion = ***Entropy***with 10 estimators, 20 min_samples_split, sqrt( of features) max_features- Slightly better F score(.60) in validation set than Gini(.58) and slighly as less overfitting.<jupyter_code>extra_trees = ExtraTreesClassifier(random_state = 10,
                                   class_weight = {0: 1, 1: 27},
                                   bootstrap = False,
                                   criterion = 'entropy',
                                   n_estimators = 10,
                                   min_samples_split = 20,
                                   max_features = "sqrt")

extra_trees.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Fit to the training set<jupyter_code># Fit to the training set
y_probs = extra_trees.predict_proba(X_train)
precs, recs, fbetas, acc_scores = plot_fbeta_recall_precision_acc(y_probs[:,1], y_train)<jupyter_output>Best Fbeta(1.5) score  0.96388 at prob decision threshold >=  0.68676<jupyter_text>Fit to the validation set<jupyter_code># Fit to the validation set
y_probs = extra_trees.predict_proba(X_val)
precs, recs, fbetas, acc_scores = plot_fbeta_recall_precision_acc(y_probs[:,1], y_val)<jupyter_output>Best Fbeta(1.5) score  0.60345 at prob decision threshold >=  0.52721<jupyter_text>2. Trying different min_samples_split from 2 to 25- Based on the results from validation set, 16 minimum number of samples to split seems to be good.<jupyter_code># Read the saved scores lists
scores_train_min = pd.read_pickle('extra_trees/scores_train_min_samples_split.pkl')
scores_val_min = pd.read_pickle('extra_trees/scores_val_min_samples_split.pkl')
scores_train_min, scores_val_min = [], [] 
min_samples_split_list = np.arange(2, 26)

for min_samples_split in min_samples_split_list:
    print(f"min_samples_split = {min_samples_split}")    
    
    extra_trees = ExtraTreesClassifier(random_state = 10,
                                       class_weight = {0: 1, 1: 27},
                                       bootstrap = False,
                                       criterion = 'entropy',
                                       n_estimators = 10,
                                       min_samples_split = min_samples_split,
                                       max_features = 'sqrt')
    
    _, scores_train_min, scores_val_min = extra_trees_result(extra_trees, X_train, y_train, X_val, y_val, scores_train_min, scores_val_min)<jupyter_output><empty_output><jupyter_text>Fit to the training set- min_samples_split doesn't seem to affect recall. Only precision will drop quite drastically as the min_samples_split increases.<jupyter_code># Fit to the training set
plot_metric_min_samples_split(min_samples_split_list, scores_train_min)<jupyter_output>Best Fbeta(1.5) score  1.00000 at min samples split = 2<jupyter_text>Fit to the validation set- As seen in the training set, precision drops even more drastically in unseen data. Min_samples_split has a direct effect on the precision probably because every tree in Extra Trees uses the same entire dataset, so the precision relies more heavily on this hyperparameter than Random Forest does.<jupyter_code># Fit to the validation set
plot_metric_min_samples_split(min_samples_split_list, scores_val_min)<jupyter_output>Best Fbeta(1.5) score  0.60320 at min samples split = 16<jupyter_text>3. Trying different n_estimators(number of trees) from 5 to 100The performance plateaus after about 30 trees. 24 trees are chosen as the best F score.<jupyter_code># Read the saved scores lists
scores_train_n_estimators = pd.read_pickle('extra_trees/scores_train_n_estimators.pkl')
scores_val_n_estimators = pd.read_pickle('extra_trees/scores_val_n_estimators.pkl')
n_estimators_list = np.arange(5, 101)

for n_estimators in n_estimators_list:
    print(f"n_estimators = {n_estimators}")    
    
    extra_trees = ExtraTreesClassifier(random_state = 10,
                                       class_weight = {0: 1, 1: 27},
                                       bootstrap = False,
                                       criterion = 'entropy',
                                       n_estimators = n_estimators,
                                       min_samples_split = 16,
                                       max_features = "sqrt")
    
    _, scores_train_n_estimators, scores_val_n_estimators = extra_trees_result(extra_trees, X_train, y_train, X_val, y_val, scores_train_n_estimators, scores_val_n_estimators)<jupyter_output><empty_output><jupyter_text>Fit to the training set- It is noticeable that the number of trees doesn't affect neither precision nor recall after about 20 trees. It is probably because every tree uses the same dataset which makes the trees overfit even with a small number of trees.<jupyter_code># Fit to the training set
plot_metric_n_estimators(n_estimators_list, scores_train_n_estimators)<jupyter_output>Best Fbeta(1.5) score  0.92992 at n_estimators = 78<jupyter_text>Fit to the validation set- Same result as training set.<jupyter_code># Fit to the validation set
plot_metric_n_estimators(n_estimators_list, scores_val_n_estimators)<jupyter_output>Best Fbeta(1.5) score  0.63227 at n_estimators = 24<jupyter_text>4. Trying different maximum number of features(candidates for splitting) from 8 to 40High number of candidate features tend to overfit by making similar trees which explains why almost the maximum parameter(39 features) is chosen in this line-search. The squared root of maximum features(about 20) seems to be good enough to prevent overfitting while maintaining about the same F score.<jupyter_code># Read the saved scores lists
scores_train_max_features = pd.read_pickle('extra_trees/scores_train_max_features.pkl')
scores_val_max_features = pd.read_pickle('extra_trees/scores_val_max_features.pkl')
for max_features in range(8, 41):
    print(f"max_features = {max_features}")    
    
    extra_trees = ExtraTreesClassifier(random_state = 10,
                                       class_weight = {0: 1, 1: 27},
                                       bootstrap = False,
                                       criterion = 'entropy',
                                       n_estimators = 24,
                                       min_samples_split = 16,
                                       max_features = max_features)
    
    _, scores_train_n_estimators, scores_val_n_estimators = extra_trees_result(extra_trees, X_train, y_train, X_val, y_val, scores_train_max_features, scores_val_max_features)<jupyter_output><empty_output><jupyter_text>Fit to the training set- Maximum number of features doesn't seem to affect recall and has a very weak positive correlation with precision.<jupyter_code># Fit to the training set
plot_metric_max_features(range(8, 41), scores_train_max_features)<jupyter_output>Best Fbeta(1.5) score  0.93292 at max features = 39<jupyter_text>Fit to the validation set<jupyter_code># Fit to the validation set
plot_metric_max_features(range(8, 41), scores_val_max_features)<jupyter_output>Best Fbeta(1.5) score  0.63749 at max features = 39<jupyter_text>Final Model of Extra Trees is...<jupyter_code>extra_trees = ExtraTreesClassifier(random_state = 10,
                                   class_weight = {0: 1, 1: 27},
                                   bootstrap = False,
                                   criterion = 'entropy',
                                   n_estimators = 24,
                                   min_samples_split = 16,
                                   max_features = "sqrt")
extra_trees.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Fit to the validation set<jupyter_code># Fit to the validation set
y_probs = extra_trees.predict_proba(X_val)[:, 1]
precs, recs, fbetas, acc_scores = plot_fbeta_recall_precision_acc(y_probs, y_val)<jupyter_output>Best Fbeta(1.5) score  0.63354 at prob decision threshold >=  0.49252<jupyter_text>Fit to the training set to check overfit.- It is still overfit after the tuning.<jupyter_code># Fit to the training set
y_probs_train = extra_trees.predict_proba(X_train)[:, 1]
precs, recs, fbetas, acc_scores = plot_fbeta_recall_precision_acc(y_probs_train, y_train)<jupyter_output>Best Fbeta(1.5) score  0.98449 at prob decision threshold >=  0.72243<jupyter_text>**Evaluation on validation set**<jupyter_code>threshold = 0.49252
pred_val = (y_probs >= threshold).astype(int)
prec, rec, fbeta, _ = precision_recall_fscore_support(y_val, pred_val, beta = 1.5, average = 'binary')
print(f"Val set: Precision = {round(prec, 5)}, Recall = {round(rec, 5)}, F_1.5 = {round(fbeta, 5)}\n")<jupyter_output>Val set: Precision = 0.61368, Recall = 0.64279, F_1.5 = 0.63354<jupyter_text>**Feature Importance**<jupyter_code>index = np.argsort(extra_trees.feature_importances_)
feature_importances = extra_trees.feature_importances_
plt.barh(range(10), feature_importances[index[-10:]])
plt.yticks(range(10), X_train.columns[index[-10:]]);<jupyter_output><empty_output><jupyter_text>The 10 important features have a very weak correlation with the target.<jupyter_code>X_train.loc[:, X_train.columns[index[-1:-10:-1]].values].corrwith(y_train)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07793373592296393, 'fraction_numerical': 0.0277460421086992, 'mean_word_length': 3.4693654266958425, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '253663', 'license': 'MIT', 'n_tokens_mistral': 4108, 'n_tokens_neox': 3723, 'n_words': 1027, 'path': 'extra_trees.ipynb', 'repo_name': 'jung-akim/fraud_detection', 'size': '12254'}