<jupyter_start><jupyter_text>Tutorial 2: AutoWoE (WhiteBox model for binary classification on tabular data)  Scorecard ![WB0](../../imgs/tutorial_whitebox_report_1.png)  Linear model ![WB1](../../imgs/tutorial_whitebox_report_2.png)  Discretization ![WB2](../../imgs/tutorial_whitebox_report_3.png)  Selection and One-dimensional analysis ![WB3](../../imgs/tutorial_whitebox_report_4.png)  Whitebox pipeline:  General parameters 0. Technical    - n_jobs    - debug1. Simple features typing and initial cleaning    1.1. Remove trash features        Medium:            - th_nan             - th_const     1.2. Typling (auto or user defined)                Critical:            - features_type (dict) {'age': 'real', 'education': 'cat', 'birth_date': (None, ("d", "wd"), ...}    1.3. Categories and datetimes encoding        Critical:            - features_type (for datetimes)        Optional:            - cat_alpha (int) - greater means more conservative encoding2. Pre selection (based on BlackBox model importances)    - Critical:        - select_type (None or int)        - imp_type (if type(select_type) is int 'perm_imt'/'feature_imp')    - Optional:        - imt_th (float) - threshold for select_type is None3. Binning (discretization)    - Critical:        - monotonic / features_monotone_constraints        - max_bin_count / max_bin_count        - min_bin_size        - cat_merge_to        - nan_merge_to    - Medium:        - force_single_split    - Optional:        - min_bin_mults        - min_gains_to_split4. WoE estimation WoE = LN( ((% 0 in bin) / (% 0 in sample)) / ((% 1 in bin) / (% 1 in sample)) ):    - Critical:        - oof_woe    - Optional:        - woe_diff_th        - n_folds (if oof_woe)5. 2nd selection stage:    5.1. One-dimentional importance        Critical:            - auc_th    5.2. VIF        Critical:            - vif_th    5.3. Partial correlations        Critical:            - pearson_th6. 3rd selection stage (model based)    - Optional:        - n_folds        - l1_base_step        - l1_exp_step    - Do not touch:        - population_size        - feature_groups_count7. Fitting the final model    - Critical:        - regularized_refit        - p_val (if not regularized_refit)        - validation (if not regularized_refit)    - Optional:        - interpreted_model        - l1_base_step (if regularized_refit)        - l1_exp_step (if regularized_refit)8. Report generation    - report_params  Imports<jupyter_code>import pandas as pd
from pandas import Series, DataFrame

import numpy as np

import os
import requests
import joblib

from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

from autowoe import AutoWoE, ReportDeco<jupyter_output><empty_output><jupyter_text>Reading the data and train/test split<jupyter_code>DATASET_DIR = '../data/'
DATASET_NAME = 'jobs_train.csv'
DATASET_FULLNAME = os.path.join(DATASET_DIR, DATASET_NAME)
DATASET_URL = 'https://raw.githubusercontent.com/sberbank-ai-lab/LightAutoML/master/examples/data/jobs_train.csv'
%%time

if not os.path.exists(DATASET_FULLNAME):
    os.makedirs(DATASET_DIR, exist_ok=True)

    dataset = requests.get(DATASET_URL).text
    with open(DATASET_FULLNAME, 'w') as output:
        output.write(dataset)
data = pd.read_csv(DATASET_FULLNAME)
data
train, test = train_test_split(data.drop('enrollee_id', axis=1), test_size=0.2, stratify=data['target'])<jupyter_output><empty_output><jupyter_text>AutoWoe: default settings<jupyter_code>auto_woe_0 = AutoWoE(interpreted_model=True,
                     monotonic=False,
                     max_bin_count=5,
                     select_type=None,
                     pearson_th=0.9,
                     auc_th=.505,
                     vif_th=10.,
                     imp_th=0,
                     th_const=32,
                     force_single_split=True,
                     th_nan=0.01,
                     th_cat=0.005,
                     auc_tol=1e-4,
                     cat_alpha=100,
                     cat_merge_to="to_woe_0",
                     nan_merge_to="to_woe_0",
                     imp_type="feature_imp",
                     regularized_refit=False,
                     p_val=0.05,
                     verbose=2
        )

auto_woe_0 = ReportDeco(auto_woe_0, )
auto_woe_0.fit(train,
               target_name="target",
              )
test_prediction = auto_woe_0.predict_proba(test)
test_prediction
roc_auc_score(test['target'].values, test_prediction)
report_params = {"output_path": "HR_REPORT_1", # папка, куда сгенерится отчет и сложатся нужные файлы
                 "report_name": "WHITEBOX REPORT",
                 "report_version_id": 1,
                 "city": "Moscow",
                 "model_aim": "Predict if candidate will work for the company",
                 "model_name": "HR model",
                 "zakazchik": "Kaggle",
                 "high_level_department": "Ai Lab",
                 "ds_name": "Btbpanda",
                 "target_descr": "Candidate will work for the company",
                 "non_target_descr": "Candidate will work for the company"}

auto_woe_0.generate_report(report_params, )<jupyter_output>No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.<jupyter_text>AutoWoE - simplier model<jupyter_code>auto_woe_1 = AutoWoE(interpreted_model=True,
                     monotonic=True,
                     max_bin_count=4,
                     select_type=None,
                     pearson_th=0.9,
                     auc_th=.505,
                     vif_th=10.,
                     imp_th=0,
                     th_const=32,
                     force_single_split=True,
                     th_nan=0.01,
                     th_cat=0.005,
                     auc_tol=1e-4,
                     cat_alpha=100,
                     cat_merge_to="to_woe_0",
                     nan_merge_to="to_woe_0",
                     imp_type="feature_imp",
                     regularized_refit=False,
                     p_val=0.05,
                     verbose=2
        )

auto_woe_1 = ReportDeco(auto_woe_1, )
auto_woe_1.fit(train,
               target_name="target",
              )
test_prediction = auto_woe_1.predict_proba(test)
test_prediction
roc_auc_score(test['target'].values, test_prediction)
report_params = {"output_path": "HR_REPORT_2", # папка, куда сгенерится отчет и сложатся нужные файлы
                 "report_name": "WHITEBOX REPORT",
                 "report_version_id": 2,
                 "city": "Moscow",
                 "model_aim": "Predict if candidate will work for the company",
                 "model_name": "HR model",
                 "zakazchik": "Kaggle",
                 "high_level_department": "Ai Lab",
                 "ds_name": "Btbpanda",
                 "target_descr": "Candidate will work for the company",
                 "non_target_descr": "Candidate will work for the company"}

auto_woe_1.generate_report(report_params, )<jupyter_output>No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.<jupyter_text>WhiteBox preset - like TabularAutoML<jupyter_code>from lightautoml.automl.presets.whitebox_presets import WhiteBoxPreset
from lightautoml import Task
task = Task('binary')
automl = WhiteBoxPreset(task)

train_pred = automl.fit_predict(train.reset_index(drop=True), roles={'target': 'target'})
test_prediction = automl.predict(test).data[:, 0]
roc_auc_score(test['target'].values, test_prediction)<jupyter_output><empty_output><jupyter_text>Serialization Важно: auto_woe_1 фактически является ReportDeco объектом (отчетом), не AutoWoE. Чтобы получить AutoWoE надо обратиться к атрибуту .model. ReportDeco не рекомендуется для использования на стадии инференса. Отчет требует целевой переменной в датасете для предсказания, так как считает метрики качества. Так же инференс из объекта-отчета намного дольше из-за собственно построения отчета.<jupyter_code>joblib.dump(auto_woe_1.model, 'model.pkl')
model = joblib.load('model.pkl')<jupyter_output><empty_output><jupyter_text>SQL inference query<jupyter_code>sql_query = model.get_sql_inference_query('global_temp.TABLE_1')
print(sql_query)<jupyter_output>SELECT
  1 / (1 + EXP(-(
    -1.111
    -0.516*WOE_TAB.city
    -0.513*WOE_TAB.city_development_index
    -0.815*WOE_TAB.company_size
    -0.398*WOE_TAB.company_type
    -0.175*WOE_TAB.experience
    -0.22*WOE_TAB.enrolled_university
    -1.24*WOE_TAB.education_level
  ))) as PROB,
  WOE_TAB.*
FROM 
    (SELECT
    CASE
      WHEN (city IS NULL OR LOWER(CAST(city AS VARCHAR(50))) = 'nan') THEN 0
      WHEN city IN ('city_100', 'city_102', 'city_103', 'city_116', 'city_149', 'city_159', 'city_160', 'city_45', 'city_46', 'city_64', 'city_71', 'city_73', 'city_83', 'city_99') THEN 0.213
      WHEN city IN ('city_104', 'city_114', 'city_136', 'city_138', 'city_16', 'city_173', 'city_23', 'city_28', 'city_36', 'city_50', 'city_57', 'city_61', 'city_65', 'city_67', 'city_75', 'city_97') THEN 1.017
      WHEN city IN ('city_11', 'city_21', 'city_74') THEN -1.455
      ELSE -0.209
    END AS city,
    CASE
      WHEN (city_development_index IS NULL OR city_development_index = 'NaN') THEN 0
   [...]<jupyter_text>Check the SQL query by PySpark<jupyter_code>from pyspark.sql import SparkSession
spark = SparkSession.builder \
                    .master("local[2]") \
                    .appName("spark-course") \
                    .config("spark.driver.memory", "512m") \
                    .getOrCreate()
sc = spark.sparkContext
spark_df = spark.read.csv("jobs_train.csv", header=True)
spark_df.createGlobalTempView("TABLE_1")
res = spark.sql(sql_query).toPandas()
res
sc.stop()
full_prediction = model.predict_proba(data)
full_prediction
(res['PROB'] - full_prediction).abs().max()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10021198689535556, 'fraction_numerical': 0.024667566005010598, 'mean_word_length': 2.5206919945725916, 'pattern_counts': {'":': 22, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '263025', 'license': 'Apache-2.0', 'n_tokens_mistral': 3539, 'n_tokens_neox': 3359, 'n_words': 768, 'path': 'examples/tutorials/Tutorial_2_WhiteBox_AutoWoE.ipynb', 'repo_name': 'Cybsloth/LightAutoML', 'size': '10378'}