<jupyter_start><jupyter_text>6장. XGBoost 하이퍼파라미터 *아래 링크를 통해 이 노트북을 주피터 노트북 뷰어(nbviewer.org)로 보거나 구글 코랩(colab.research.google.com)에서 실행할 수 있습니다.*      주피터 노트북 뷰어로 보기        구글 코랩(Colab)에서 실행하기<jupyter_code># 노트북이 코랩에서 실행 중인지 체크합니다.
import sys
if 'google.colab' in sys.modules:
    !pip install -q --upgrade xgboost
    !wget -q https://raw.githubusercontent.com/rickiepark/handson-gb/main/Chapter06/heart_disease.csv
# 경고 끄기
import warnings
warnings.filterwarnings('ignore')

import xgboost as xgb
xgb.set_config(verbosity=0)<jupyter_output><empty_output><jupyter_text>데이터와 기본 모델 준비하기 심장 질환 데이터셋<jupyter_code>import pandas as pd
df = pd.read_csv('heart_disease.csv')
df.head()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 303 entries, 0 to 302
Data columns (total 14 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   age       303 non-null    int64  
 1   sex       303 non-null    int64  
 2   cp        303 non-null    int64  
 3   trestbps  303 non-null    int64  
 4   chol      303 non-null    int64  
 5   fbs       303 non-null    int64  
 6   restecg   303 non-null    int64  
 7   thalach   303 non-null    int64  
 8   exang     303 non-null    int64  
 9   oldpeak   303 non-null    float64
 10  slope     303 non-null    int64  
 11  ca        303 non-null    int64  
 12  thal      303 non-null    int64  
 13  target    303 non-null    int64  
dtypes: float64(1), int64(13)
memory usage: 33.3 KB<jupyter_text>XGBClassifier<jupyter_code>from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score
# 데이터를 X와 y로 나눕니다.
X = df.iloc[:, :-1]
y = df.iloc[:, -1]
model = XGBClassifier(booster='gbtree', objective='binary:logistic')
# cross_val_score를 임포트합니다.
from sklearn.model_selection import cross_val_score

# numpy를 임포트합니다.
import numpy as np

# 교차 검증 점수를 구합니다.
scores = cross_val_score(model, X, y, cv=5)

# 정확도를 출력합니다.
print('정확도:', np.round(scores, 2))

# 정확도 평균을 출력합니다.
print('정확도 평균: %0.2f' % (scores.mean()))<jupyter_output>정확도: [0.84 0.85 0.82 0.8  0.77]
정확도 평균: 0.81<jupyter_text>StratifiedKFold<jupyter_code># GridSearchCV를 임포트합니다.
from sklearn.model_selection import StratifiedKFold

kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=2)
print('전체 데이터의 클래스 분포:', np.bincount(y))
print()

for split_no, (train_idx, test_idx) in enumerate(kfold.split(X, y)):
    print('{}번째 훈련 폴드:'.format(split_no+1), np.bincount(y[train_idx]))
    print('{}번째 검증 폴드:'.format(split_no+1), np.bincount(y[test_idx]))
    print()<jupyter_output>전체 데이터의 클래스 분포: [138 165]

1번째 훈련 폴드: [110 132]
1번째 검증 폴드: [28 33]

2번째 훈련 폴드: [110 132]
2번째 검증 폴드: [28 33]

3번째 훈련 폴드: [110 132]
3번째 검증 폴드: [28 33]

4번째 훈련 폴드: [111 132]
4번째 검증 폴드: [27 33]

5번째 훈련 폴드: [111 132]
5번째 검증 폴드: [27 33]<jupyter_text>기본 모델<jupyter_code># 교차 검증 점수를 계산합니다.
scores = cross_val_score(model, X, y, cv=kfold)

# 정확도를 출력합니다.
print('정확도:', np.round(scores, 2))

# 정확도 평균을 출력합니다.
print('정확도 평균: %0.2f' % (scores.mean()))<jupyter_output>정확도: [0.72 0.82 0.75 0.8  0.82]
정확도 평균: 0.78<jupyter_text>GridSearchCV와 RandomizedSearchCV 사용하기<jupyter_code># GridSearchCV를 임포트합니다.
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=2)
def grid_search(params, random=False): 

    xgb = XGBClassifier(booster='gbtree', objective='binary:logistic', 
                        random_state=2, verbosity=0, use_label_encoder=False)
    
    kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=2)
    
    if random:
        grid = RandomizedSearchCV(xgb, params, cv=kfold, n_iter=20, 
                                  n_jobs=-1, random_state=2)
    else:
        grid = GridSearchCV(xgb, params, cv=kfold, n_jobs=-1)
    
    # X와 y에서 하이퍼파라미터 튜닝을 수행합니다.
    grid.fit(X, y)

    # 최상의 매개변수를 추출합니다.
    best_params = grid.best_params_

    # 최상의 매개변수를 출력합니다.
    print("최상의 매개변수:", best_params)
    
    # 최상의 점수를 추출합니다.
    best_score = grid.best_score_

    # 최상의 점수를 출력합니다.
    print("최상의 점수: {:.5f}".format(best_score))<jupyter_output><empty_output><jupyter_text>XGBoost 하이퍼파라미터 튜닝 n_estimators<jupyter_code>grid_search(params={'n_estimators':[100, 200, 400, 800]})<jupyter_output>최상의 매개변수: {'n_estimators': 100}
최상의 점수: 0.78235<jupyter_text>learning_rate<jupyter_code>grid_search(params={'learning_rate':[0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5]})<jupyter_output>최상의 매개변수: {'learning_rate': 0.05}
최상의 점수: 0.79585<jupyter_text>max_depth<jupyter_code>grid_search(params={'max_depth':[2, 3, 5, 6, 8]})<jupyter_output>최상의 매개변수: {'max_depth': 2}
최상의 점수: 0.79902<jupyter_text>gamma<jupyter_code>grid_search(params={'gamma':[0, 0.01, 0.1, 0.5, 1, 2]})<jupyter_output>최상의 매개변수: {'gamma': 0.5}
최상의 점수: 0.79574<jupyter_text>min_child_weight<jupyter_code>grid_search(params={'min_child_weight':[1, 2, 3, 4, 5]})<jupyter_output>최상의 매개변수: {'min_child_weight': 5}
최상의 점수: 0.81219<jupyter_text>subsample<jupyter_code>grid_search(params={'subsample':[0.5, 0.7, 0.8, 0.9, 1]})<jupyter_output>최상의 매개변수: {'subsample': 0.8}
최상의 점수: 0.79579<jupyter_text>colsample_bytree<jupyter_code>grid_search(params={'colsample_bytree':[0.5, 0.7, 0.8, 0.9, 1]})<jupyter_output>최상의 매개변수: {'colsample_bytree': 0.5}
최상의 점수: 0.80552<jupyter_text>조기 종료 적용 eval_set과 eval_metric<jupyter_code># train_test_split 함수를 임포트합니다.
from sklearn.model_selection import train_test_split

# 훈련 세트와 테스트 세트로 나눕니다.
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=2)
model = XGBClassifier(booster='gbtree', objective='binary:logistic')
eval_set = [(X_test, y_test)]
eval_metric='error'
model.fit(X_train, y_train, eval_metric=eval_metric, 
          eval_set=eval_set)

# 테스트 세트에 대한 예측을 만듭니다.
y_pred = model.predict(X_test)
# 예측을 평가합니다.
accuracy = accuracy_score(y_test, y_pred)
print("정확도: %.2f%%" % (accuracy * 100.0))<jupyter_output>[0]	validation_0-error:0.15789
[1]	validation_0-error:0.10526
[2]	validation_0-error:0.11842
[3]	validation_0-error:0.13158
[4]	validation_0-error:0.11842
[5]	validation_0-error:0.14474
[6]	validation_0-error:0.14474
[7]	validation_0-error:0.14474
[8]	validation_0-error:0.14474
[9]	validation_0-error:0.14474
[10]	validation_0-error:0.14474
[11]	validation_0-error:0.15789
[12]	validation_0-error:0.15789
[13]	validation_0-error:0.17105
[14]	validation_0-error:0.17105
[15]	validation_0-error:0.17105
[16]	validation_0-error:0.15789
[17]	validation_0-error:0.17105
[18]	validation_0-error:0.15789
[19]	validation_0-error:0.17105
[20]	validation_0-error:0.17105
[21]	validation_0-error:0.17105
[22]	validation_0-error:0.18421
[23]	validation_0-error:0.18421
[24]	validation_0-error:0.17105
[25]	validation_0-error:0.18421
[26]	validation_0-error:0.18421
[27]	validation_0-error:0.18421
[28]	validation_0-error:0.18421
[29]	validation_0-error:0.18421
[30]	validation_0-error:0.18421
[31]	validation_0-[...]<jupyter_text>early_stopping_rounds<jupyter_code>model = XGBClassifier(booster='gbtree', objective='binary:logistic')
eval_set = [(X_test, y_test)]
eval_metric="error"
model.fit(X_train, y_train, eval_metric=eval_metric, eval_set=eval_set, 
          early_stopping_rounds=10, verbose=True)
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print("정확도: %.2f%%" % (accuracy * 100.0))
model = XGBClassifier(n_estimators=5000)
eval_set = [(X_test, y_test)]
eval_metric="error"
model.fit(X_train, y_train, eval_metric=eval_metric, eval_set=eval_set, 
          early_stopping_rounds=100)
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print("정확도: %.2f%%" % (accuracy * 100.0))<jupyter_output>[0]	validation_0-error:0.15789
[1]	validation_0-error:0.10526
[2]	validation_0-error:0.11842
[3]	validation_0-error:0.13158
[4]	validation_0-error:0.11842
[5]	validation_0-error:0.14474
[6]	validation_0-error:0.14474
[7]	validation_0-error:0.14474
[8]	validation_0-error:0.14474
[9]	validation_0-error:0.14474
[10]	validation_0-error:0.14474
[11]	validation_0-error:0.15789
[12]	validation_0-error:0.15789
[13]	validation_0-error:0.17105
[14]	validation_0-error:0.17105
[15]	validation_0-error:0.17105
[16]	validation_0-error:0.15789
[17]	validation_0-error:0.17105
[18]	validation_0-error:0.15789
[19]	validation_0-error:0.17105
[20]	validation_0-error:0.17105
[21]	validation_0-error:0.17105
[22]	validation_0-error:0.18421
[23]	validation_0-error:0.18421
[24]	validation_0-error:0.17105
[25]	validation_0-error:0.18421
[26]	validation_0-error:0.18421
[27]	validation_0-error:0.18421
[28]	validation_0-error:0.18421
[29]	validation_0-error:0.18421
[30]	validation_0-error:0.18421
[31]	validation_0-[...]<jupyter_text>하이퍼파라미터 결합 하나씩 하이퍼파라미터 조사하기 n_estimators<jupyter_code>grid_search(params={'n_estimators':[2, 25, 50, 75, 100]})<jupyter_output>최상의 매개변수: {'n_estimators': 50}
최상의 점수: 0.78907<jupyter_text>max_depth<jupyter_code>grid_search(params={'max_depth':[1, 2, 3, 4, 6, 7, 8], 
                    'n_estimators':[50]})
grid_search(params={'max_depth':[1, 2, 3, 4, 6, 7, 8], 
                    'n_estimators':[2, 50, 100]})<jupyter_output>최상의 매개변수: {'max_depth': 1, 'n_estimators': 50}
최상의 점수: 0.83869<jupyter_text>learning_rate<jupyter_code>grid_search(params={'learning_rate':[0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5], 
                    'max_depth':[1], 'n_estimators':[50]})<jupyter_output>최상의 매개변수: {'learning_rate': 0.3, 'max_depth': 1, 'n_estimators': 50}
최상의 점수: 0.83869<jupyter_text>min_child_weight<jupyter_code>grid_search(params={'min_child_weight':[1, 2, 3, 4, 5], 
                    'max_depth':[1], 'n_estimators':[50]})<jupyter_output>최상의 매개변수: {'max_depth': 1, 'min_child_weight': 1, 'n_estimators': 50}
최상의 점수: 0.83869<jupyter_text>subsample<jupyter_code>grid_search(params={'subsample':[0.5, 0.6, 0.7, 0.8, 0.9, 1],
                    'max_depth':[1], 'n_estimators':[50]})
grid_search(params={'subsample':[0.5, 0.6, 0.7, 0.8, 0.9, 1], 
                    'min_child_weight':[1, 2, 3, 4, 5], 
                    'learning_rate':[0.1, 0.2, 0.3, 0.4, 0.5], 
                    'max_depth':[1, 2, 3, 4, 5], 
                    'n_estimators':[2]})<jupyter_output>최상의 매개변수: {'learning_rate': 0.5, 'max_depth': 2, 'min_child_weight': 4, 'n_estimators': 2, 'subsample': 0.9}
최상의 점수: 0.81224<jupyter_text>하이퍼파라미터 조정<jupyter_code>grid_search(params={'subsample':[0.5, 0.6, 0.7, 0.8, 0.9, 1], 
                    'min_child_weight':[1, 2, 3, 4, 5], 
                    'learning_rate':[0.1, 0.2, 0.3, 0.4, 0.5], 
                    'max_depth':[1, 2, 3, 4, 5, None], 
                    'n_estimators':[2, 25, 50, 75, 100]}, random=True)<jupyter_output>최상의 매개변수: {'subsample': 0.6, 'n_estimators': 25, 'min_child_weight': 4, 'max_depth': 4, 'learning_rate': 0.5}
최상의 점수: 0.82208<jupyter_text>colsample colsample_bytree<jupyter_code>grid_search(params={'colsample_bytree':[0.5, 0.6, 0.7, 0.8, 0.9, 1],
                    'max_depth':[1], 'n_estimators':[50]})<jupyter_output>최상의 매개변수: {'colsample_bytree': 1, 'max_depth': 1, 'n_estimators': 50}
최상의 점수: 0.83869<jupyter_text>colsample_bylevel<jupyter_code>grid_search(params={'colsample_bylevel':[0.5, 0.6, 0.7, 0.8, 0.9, 1],
                    'max_depth':[1], 'n_estimators':[50]})<jupyter_output>최상의 매개변수: {'colsample_bylevel': 1, 'max_depth': 1, 'n_estimators': 50}
최상의 점수: 0.83869<jupyter_text>colsample_bynode<jupyter_code>grid_search(params={'colsample_bynode':[0.5, 0.6, 0.7, 0.8, 0.9, 1], 
                    'colsample_bylevel':[0.5, 0.6, 0.7, 0.8, 0.9, 1], 
                    'colsample_bytree':[0.5, 0.6, 0.7, 0.8, 0.9, 1], 
                    'max_depth':[1], 'n_estimators':[50]})<jupyter_output>최상의 매개변수: {'colsample_bylevel': 0.6, 'colsample_bynode': 0.7, 'colsample_bytree': 0.8, 'max_depth': 1, 'n_estimators': 50}
최상의 점수: 0.84852<jupyter_text>gamma<jupyter_code>grid_search(params={'gamma':[0, 0.01, 0.05, 0.1, 0.5, 1, 2, 3], 
                    'colsample_bylevel':[0.9], 
                    'colsample_bytree':[0.8], 
                    'colsample_bynode':[0.5], 
                    'max_depth':[1], 'n_estimators':[50]})<jupyter_output>최상의 매개변수: {'colsample_bylevel': 0.9, 'colsample_bynode': 0.5, 'colsample_bytree': 0.8, 'gamma': 0, 'max_depth': 1, 'n_estimators': 50}
최상의 점수: 0.83536
{'fraction_non_alphanumeric': 0.15946789292166202, 'fraction_numerical': 0.10477910987025785, 'mean_word_length': 4.322989510489511, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '389852', 'license': 'MIT', 'n_tokens_mistral': 6605, 'n_tokens_neox': 6467, 'n_words': 954, 'path': 'Chapter06/XGBoost_Hyperparameters.ipynb', 'repo_name': 'rickiepark/handson-gb', 'size': '12178'}