<jupyter_start><jupyter_text>[Kaggle Link](https://www.kaggle.com/c/santander-customer-satisfaction)  데이터 전처리<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib

cust_df = pd.read_csv("./train_santander.csv", encoding='latin-1')
print(f"data shape: {cust_df.shape}")
cust_df.head()
# 데이터 타입 확인
cust_df.info()
print(cust_df['TARGET'].value_counts())
# 불만족 건수 비율 확인
unsatisfied_cnt = cust_df[cust_df['TARGET'] == 1]['TARGET'].count()
total_cnt = cust_df['TARGET'].count()

print(f"불만족 건수 비율: {(unsatisfied_cnt / total_cnt):.2f}")
# 피쳐 분포도 확인
cust_df.describe()
print(cust_df['var3'].value_counts()[:10])<jupyter_output>2         74165
 8           138
-999999      116
 9           110
 3           108
 1           105
 13           98
 7            97
 4            86
 12           85
Name: var3, dtype: int64<jupyter_text>-999999 는 타 값들과 너무 큰 차이가 남 가장많은 2로 대체<jupyter_code># var3 피쳐 값 대체 및 ID 피쳐 드롭
cust_df['var3'].replace(-999999, 2, inplace=True)
cust_df.drop('ID', axis=1, inplace=True)
# 피쳐 세트와 레이블 세트 분리
X_features = cust_df.iloc[:, :-1]
y_labels = cust_df.iloc[:,-1]
print(f"피쳐 데이터 shape: {X_features.shape}")<jupyter_output>피쳐 데이터 shape: (76020, 369)<jupyter_text>학습/테스트 데이터 분할<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X_features, y_labels,
                                                    test_size=0.3, random_state=216, 
                                                    stratify = y_labels) # stratify로 학습 및 테스트 세트 레이블 값 비율을 맞춰줌
train_cnt = y_train.count()
test_cnt = y_test.count()
print(f"학습 세트 shape:{X_train.shape}, 테스트 세트 shape:{X_test.shape}\n")

print(f"학습 세트 레이블 값 분포 비율:\n{y_train.value_counts()/train_cnt}\n")
print(f"테스트 세트 레이블 값 분포 비율:\n{y_test.value_counts()/test_cnt}")<jupyter_output>학습 세트 shape:(53214, 369), 테스트 세트 shape:(22806, 369)

학습 세트 레이블 값 분포 비율:
0    0.960424
1    0.039576
Name: TARGET, dtype: float64

테스트 세트 레이블 값 분포 비율:
0    0.960449
1    0.039551
Name: TARGET, dtype: float64<jupyter_text>XGBoost 모델 학습 수행 evaluation은 학습이 제대로 되고 있는지 방향성을 알려준다. 즉 학습데이터와 검증데이터로 나누고 학습을 수행하면서 평가를 검증데이터로 계속 진행한다.보통은 evaluation 데이터를 학습 데이터의 일부를 쪼개서 별도로 구성한다.테스트 데이터는 아예 학습/검증 데이터가 아닌 다른 데이터를 사용한다.<jupyter_code>from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score

xgb_clf = XGBClassifier(n_estimators=500, random_state=216)

xgb_clf.fit(X_train, y_train, early_stopping_rounds=100,
            eval_metric="auc", eval_set=[(X_train, y_train), (X_test, y_test)])
xgb_roc_score = roc_auc_score(y_test, xgb_clf.predict_proba(X_test)[:,1], average='macro')
print(f"ROC AUC: {xgb_roc_score:.4f}")<jupyter_output>E:\anaconda\lib\site-packages\xgboost\sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
  warnings.warn(label_encoder_deprecation_msg, UserWarning)<jupyter_text>하이퍼 파라미터 튜닝<jupyter_code>from sklearn.model_selection import GridSearchCV

# 개인 PC 역량 한계로 인해 n_estimator=100으로 설정
xgb_clf = XGBClassifier(n_estimators=100)

params = {'max_depth':[5,7], 'min_child_weight':[1,3], 'colsample_bytree':[0.5,0.75]}

# 하이퍼 파라미터 테스트 수행속도를 위해 cv지정하지않음
gridcv = GridSearchCV(xgb_clf, param_grid=params)
gridcv.fit(X_train, y_train, early_stopping_rounds=30, eval_metric='auc',
          eval_set=[(X_train, y_train), (X_test, y_test)])

print(f"최적 파라미터:{gridcv.best_params_}")

xgb_roc_score = roc_auc_score(y_test, gridcv.predict_proba(X_test)[:,1], average='macro')
print(f"ROC AUC: {xgb_roc_score:.4f}")<jupyter_output>E:\anaconda\lib\site-packages\xgboost\sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
  warnings.warn(label_encoder_deprecation_msg, UserWarning)<jupyter_text>좋은 알고리즘으로는 하이퍼파라미터 엔지니어링으로 예측성능에 큰 영향을 미치지 못함피쳐 엔지니어링이 훨씬 도움됨<jupyter_code>xgb_clf = XGBClassifier(n_estimator=100, random_state=216, colsample_bytree=0.75,
                        max_depth=5, min_child_weight=3)

xgb_clf.fit(X_train, y_train, early_stopping_rounds=200, eval_metric="auc",
            eval_set=[(X_train, y_train), (X_test, y_test)])
from xgboost import plot_importance
import matplotlib.pyplot as plt
%matplotlib inline

fig, ax = plt.subplots(1,1, figsize=(10,10))
plot_importance(xgb_clf, ax=ax, max_num_features=20, height=0.4)<jupyter_output><empty_output><jupyter_text>LightGBM 모델 학습<jupyter_code>from lightgbm import LGBMClassifier

lgbm_clf = LGBMClassifier(n_estimators=500)

evals = [(X_test, y_test)]
lgbm_clf.fit(X_train, y_train, early_stopping_rounds=100, eval_metric='auc', eval_set=evals, verbose=True)

lgbm_roc_score = roc_auc_score(y_test, lgbm_clf.predict_proba(X_test)[:,1], average='macro')
print(f"ROC AUC: {lgbm_roc_score:.4f}")<jupyter_output>[1]	valid_0's auc: 0.819645	valid_0's binary_logloss: 0.15865
Training until validation scores don't improve for 100 rounds
[2]	valid_0's auc: 0.8207	valid_0's binary_logloss: 0.153765
[3]	valid_0's auc: 0.825	valid_0's binary_logloss: 0.150353
[4]	valid_0's auc: 0.830254	valid_0's binary_logloss: 0.147636
[5]	valid_0's auc: 0.829788	valid_0's binary_logloss: 0.145639
[6]	valid_0's auc: 0.830257	valid_0's binary_logloss: 0.143931
[7]	valid_0's auc: 0.831207	valid_0's binary_logloss: 0.142572
[8]	valid_0's auc: 0.83163	valid_0's binary_logloss: 0.141435
[9]	valid_0's auc: 0.832528	valid_0's binary_logloss: 0.140438
[10]	valid_0's auc: 0.833001	valid_0's binary_logloss: 0.139556
[11]	valid_0's auc: 0.833344	valid_0's binary_logloss: 0.138965
[12]	valid_0's auc: 0.832629	valid_0's binary_logloss: 0.138506
[13]	valid_0's auc: 0.833735	valid_0's binary_logloss: 0.13803
[14]	valid_0's auc: 0.83433	valid_0's binary_logloss: 0.137512
[15]	valid_0's auc: 0.834527	valid_0's binary_logloss: 0.137[...]<jupyter_text>하이퍼 파라미터 튜닝<jupyter_code>from sklearn.model_selection import GridSearchCV

LGBM_clf = LGBMClassifier(n_estimators=200)

params={'num_leaves':[32,65],
        'max_depth': [128, 160],
        'min_chid_samples':[60,100],
        'subsample':[0.8, 1]}

# 하이퍼 파라미터 수행속도 향상을 위해 cv지정 하지 않음
gridcv = GridSearchCV(lgbm_clf, param_grid=params)
gridcv.fit(X_train, y_train, early_stopping_rounds=30, eval_metric='auc',
           eval_set=[(X_train, y_train), (X_test, y_test)])
print("최적 파라미터: ", gridcv.best_params_)
lgbm_roc_score = roc_auc_score(y_test, gridcv.predict_proba(X_test)[:,1], average='macro')
print(f"ROC-AUC: {lgbm_roc_score:.4f}")
lgbm_clf = LGBMClassifier(n_estimators=100, num_leaves=65 , subsample=0.8 , min_child_samples=60 , max_depth=128 )

evals = [(X_test, y_test)]
lgbm_clf.fit(X_train, y_train, early_stopping_rounds=100, eval_metric="auc", eval_set=evals,
            verbose=True)

lgbm_roc_score = roc_auc_score(y_test, lgbm_clf.predict_proba(X_test)[:,1], average='macro')
print(f"ROC AUC : {lgbm_roc_score:.4f}")<jupyter_output>[1]	valid_0's auc: 0.817874	valid_0's binary_logloss: 0.158281
Training until validation scores don't improve for 100 rounds
[2]	valid_0's auc: 0.826083	valid_0's binary_logloss: 0.153417
[3]	valid_0's auc: 0.828801	valid_0's binary_logloss: 0.150004
[4]	valid_0's auc: 0.831244	valid_0's binary_logloss: 0.147217
[5]	valid_0's auc: 0.831472	valid_0's binary_logloss: 0.145165
[6]	valid_0's auc: 0.832222	valid_0's binary_logloss: 0.143428
[7]	valid_0's auc: 0.834378	valid_0's binary_logloss: 0.142141
[8]	valid_0's auc: 0.835589	valid_0's binary_logloss: 0.141085
[9]	valid_0's auc: 0.83627	valid_0's binary_logloss: 0.140047
[10]	valid_0's auc: 0.836448	valid_0's binary_logloss: 0.139261
[11]	valid_0's auc: 0.83668	valid_0's binary_logloss: 0.138559
[12]	valid_0's auc: 0.836518	valid_0's binary_logloss: 0.137976
[13]	valid_0's auc: 0.836854	valid_0's binary_logloss: 0.137493
[14]	valid_0's auc: 0.836512	valid_0's binary_logloss: 0.137159
[15]	valid_0's auc: 0.83707	valid_0's binary_logloss:[...]
{'fraction_non_alphanumeric': 0.12369023244610382, 'fraction_numerical': 0.09249668794411658, 'mean_word_length': 5.192393736017897, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '183715', 'license': 'MIT', 'n_tokens_mistral': 4363, 'n_tokens_neox': 4228, 'n_words': 757, 'path': 'Santander Customer Satisfaction/Santander_Customer_Satisfaction.ipynb', 'repo_name': 'kimbyeolhee/Kaggle', 'size': '8303'}