<jupyter_start><jupyter_text>Feature engineering------------------------------This example shows how to use automated feature generation to improve a model's performance.The data used is a variation on the [Australian weather dataset](https://www.kaggle.com/jsphyg/weather-dataset-rattle-package) from Kaggle. You can download it from [here](https://github.com/tvdboom/ATOM/blob/master/examples/datasets/weatherAUS.csv). The goal of this dataset is to predict whether or not it will rain tomorrow training a binary classifier on target `RainTomorrow`.  Load the data<jupyter_code># Import packages
import pandas as pd
from atom import ATOMClassifier
# Load data
X = pd.read_csv("./datasets/weatherAUS.csv")

# Let's have a look
X.head()<jupyter_output><empty_output><jupyter_text>Run the pipeline<jupyter_code># Initialize atom and apply data cleaning
atom = ATOMClassifier(X, n_rows=1e4, test_size=0.2, verbose=0, random_state=1)
atom.clean()
atom.impute(strat_num="knn", strat_cat="remove", max_nan_rows=0.8)
atom.encode(max_onehot=10, frac_to_other=0.04)
atom.verbose = 2  # Increase verbosity to see the output

# Let's see how a LightGBM model performs
atom.run('LGB', metric='auc')<jupyter_output>Training ========================= >>
Models: LGB
Metric: roc_auc

Results for LightGBM:
Fit ---------------------------------------------
Train evaluation --> roc_auc: 0.9854
Test evaluation --> roc_auc: 0.8788
Time elapsed: 0.275s
-------------------------------------------------
Total time: 0.275s


Final results ==================== >>
Duration: 0.276s
-------------------------------------
LightGBM --> roc_auc: 0.8788<jupyter_text>Deep Feature Synthesis<jupyter_code># Since we are going to compare different datasets,
# we need to create separate branches
atom.branch = "dfs"
# Create 50 new features using DFS
atom.feature_generation("dfs", n_features=50, operators=["add", "sub", "log"])
# The warnings warn us that some operators created missing values!
# We can see the columns with missing values using the nans attribute
atom.nans
# Turn off warnings in the future
atom.warnings = False

# Impute the data again to get rid of the missing values
atom.impute(strat_num="knn", strat_cat="remove", max_nan_rows=0.8)
# 50 new features may be to much...
# Let's check for multicollinearity and use RFECV to reduce the number
atom.feature_selection(
    strategy="RFECV",
    solver="LGB",
    n_features=30,
    scoring="auc",
    max_correlation=0.98,
)
# The collinear attribute shows what features were removed due to multicollinearity
atom.collinear
# After applying RFECV, we can plot the score per number of features
atom.plot_rfecv()
# Let's see how the model performs now
# Add a tag to the model's acronym to not overwrite previous LGB
atom.run("LGB_dfs")<jupyter_output>Training ========================= >>
Models: LGB_dfs
Metric: roc_auc

Results for LightGBM:
Fit ---------------------------------------------
Train evaluation --> roc_auc: 0.9929
Test evaluation --> roc_auc: 0.8711
Time elapsed: 0.576s
-------------------------------------------------
Total time: 0.576s


Final results ==================== >>
Duration: 0.577s
-------------------------------------
LightGBM --> roc_auc: 0.8711<jupyter_text>Genetic Feature Generation<jupyter_code># Create another branch for the genetic features
# Split form master to avoid the dfs features
atom.branch = "gfg_from_master"
# Create new features using Genetic Programming
atom.feature_generation(
    strategy='GFG',
    n_features=20,
    generations=10,
    population=2000,
)
# We can see the feature's fitness and description through the genetic_features attribute
atom.genetic_features
# Fit the model again
atom.run("LGB_gfg", metric="auc")<jupyter_output>Training ========================= >>
Models: LGB_gfg
Metric: roc_auc

Results for LightGBM:
Fit ---------------------------------------------
Train evaluation --> roc_auc: 0.9901
Test evaluation --> roc_auc: 0.8734
Time elapsed: 0.421s
-------------------------------------------------
Total time: 0.421s


Final results ==================== >>
Duration: 0.422s
-------------------------------------
LightGBM --> roc_auc: 0.8734<jupyter_text>Analyze results<jupyter_code># Use atom's plots to compare the three models
atom.palette = "Paired"
atom.plot_roc(dataset="both")
atom.reset_aesthetics()
# For busy plots it might be useful to use a canvas
with atom.canvas(1, 3, figsize=(20, 8)):
    atom.lgb.plot_feature_importance(show=10, title="LGB")
    atom.lgb_dfs.plot_feature_importance(show=10, title="LGB + DFS")
    atom.lgb_gfg.plot_feature_importance(show=10, title="LGB + GFG")
# We can check the feature importance with other plots as well
atom.plot_permutation_importance(models=["LGB_DFS", "LGB_GFG"], show=10)
atom.LGB_gfg.decision_plot(index=(0, 10), show=15)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.20317984720214743, 'fraction_numerical': 0.027668800330373736, 'mean_word_length': 3.403636363636364, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '142041', 'license': 'MIT', 'n_tokens_mistral': 1575, 'n_tokens_neox': 1440, 'n_words': 447, 'path': 'examples/feature_engineering.ipynb', 'repo_name': 'tvdboom/ATOM', 'size': '4843'}