<jupyter_start><jupyter_text>I'm now repeating code a lot here, bad! Will need to write bits of this<jupyter_code>path_to_data = os.path.abspath(os.path.join(os.getcwd(),
                                            "..",
                                            "data/processed/"
                                           ))

data_dict = train.load_processed_data(file_path=path_to_data)
X = data_dict["X_train"]
y = data_dict["y_train"]
X_train, X_test, y_train, y_test = sel.train_test_split(X,
                                                    y,
                                                    test_size=0.3,
                                                    random_state=42,
                                                    shuffle=True,
                                                    stratify=y)
X_train_smol, X_test_smol, y_train_smol, y_test_smol = sel.train_test_split(X,
                                                    y,
                                                    test_size=0.09,
                                                    train_size=0.21,
                                                    random_state=42,
                                                    shuffle=True,
                                                    stratify=y)
et_clf = ensemble.ExtraTreesClassifier(  random_state=42,
                                         verbose=True)
et_params = {"n_estimators": [100,200,500,1000,1500],
            }
grid_search = sel.GridSearchCV(estimator=et_clf, 
                               param_grid=et_params,
                               scoring = "f1_macro",
                               n_jobs = 2,
                               cv=3,
                               verbose = 10
                              )
grid_search.fit(X_train_smol, y_train_smol)
grid_search.best_params_
grid_search.best_score_<jupyter_output><empty_output><jupyter_text>the biggest model is bestest.<jupyter_code>et_results_df = pd.DataFrame(grid_search.cv_results_)
sb.lineplot(x="param_n_estimators", y="mean_test_score",data=et_results_df)<jupyter_output><empty_output><jupyter_text>Extra trees appear to outperform random forest, so this means our data is noisey and the greedy approach that random forests take to feature selection for fitting the trees is not working as well as a random search.We also get relatively more improvement as we increase size than random forest models.lets try looking at these around 1000<jupyter_code>et_params = {"n_estimators": [x for x in range(800,1250,50)],
            }
grid_search = sel.GridSearchCV(estimator=et_clf, 
                               param_grid=et_params,
                               scoring = "f1_macro",
                               n_jobs = 2,
                               cv=3,
                               verbose = 10
                              )
grid_search.fit(X_train_smol, y_train_smol)
et_results_df = pd.DataFrame(grid_search.cv_results_)
sb.lineplot(x="param_n_estimators", y="mean_test_score",data=et_results_df)<jupyter_output><empty_output><jupyter_text>looks like noise mostly. Will stick with around 800<jupyter_code>et_params = {"n_estimators": [600,700,800,900,1000],
             "criterion": ["gini", "entropy"]
            }
grid_search = sel.GridSearchCV(estimator=et_clf, 
                               param_grid=et_params,
                               scoring = "f1_macro",
                               n_jobs = 2,
                               cv=3,
                               verbose = 10
                              )
grid_search.fit(X_train_smol, y_train_smol)
grid_search.best_params_
grid_search.best_score_
et_results_df = pd.DataFrame(grid_search.cv_results_)
et_results_df
sb.lineplot(x="param_n_estimators", 
            y="mean_test_score",
            hue = "param_criterion",
            data=et_results_df)<jupyter_output><empty_output><jupyter_text>not really any difference, we go with gini as it is less computationally intensive<jupyter_code>et_features_params = {"n_estimators": [600,700,800,900,1000],
             "criterion": ["gini"],
             "max_features": ["auto", "sqrt", "log2", None]
            }


grid_search = sel.GridSearchCV(estimator=et_clf, 
                               param_grid=et_features_params,
                               scoring = "f1_macro",
                               n_jobs = 2,
                               cv=3,
                               verbose = 10
                              )
grid_search.fit(X_train_smol, y_train_smol)
grid_search.best_params_
grid_search.best_score_<jupyter_output><empty_output><jupyter_text>not making any changes, looks like we top out at 0.936ish. Bit better than random forests, nearly as good as the KNN, but not as good as the SVMs<jupyter_code>et_clf = ensemble.ExtraTreesClassifier(criterion="gini",
                                       max_features="auto",
                                       n_estimators=400,
                                       random_state=42,
                                       verbose=True,
                                       n_jobs=1)
et_cv_results = sel.cross_validate(et_clf,
                                   X_train,
                                   y_train,
                                   cv=3,
                                   n_jobs=1,
                                   scoring="f1_macro",
                                   return_train_score=True,
                                   return_estimator=True,
                                   verbose=True
                                   )
et_cv_results["test_score"]
np.mean(et_cv_results["test_score"])
et_clf.fit(X_train, y_train)
y_pred = et_clf.predict(X_test)
metrics.f1_score(y_test, y_pred, average="macro")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07553648068669527, 'fraction_numerical': 0.019570815450643778, 'mean_word_length': 1.3341346153846154, 'pattern_counts': {'":': 7, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 44, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '283299', 'license': 'FTL', 'n_tokens_mistral': 1658, 'n_tokens_neox': 1540, 'n_words': 274, 'path': 'notebooks/07-er-training-extra_trees.ipynb', 'repo_name': 'edd-r/minst-ensemble', 'size': '5825'}