<jupyter_start><jupyter_text>練習題 I: GridSearch練習說明：在08_M8課程中，我們使用LightGBM 來示範 GridSearch(網格搜索法)，對Book_Purchased進行建模。本練習題中，希望各位學員能以Xgboost來進行葡萄牙銀行的電話營銷成功與否的預測，並以GridSearch對Xgboost中的參數進行最優化的自動搜尋。Ref: https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8  資料下載：---https://raw.githubusercontent.com/madmashup/targeted-marketing-predictive-engine/master/banking.csv該數據集來自 UCI 機器學習存儲庫，是關於葡萄牙銀行的電話營銷活動。分類目標是預測客戶是否會訂閱定期存款（變量 y：0/1）。<jupyter_code>!wget https://raw.githubusercontent.com/madmashup/targeted-marketing-predictive-engine/master/banking.csv<jupyter_output>--2022-01-20 11:40:33--  https://raw.githubusercontent.com/madmashup/targeted-marketing-predictive-engine/master/banking.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 192.168.127.12, 172.16.58.3, 192.168.3.11, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4882918 (4.7M) [text/plain]
Saving to: ‘banking.csv.1’

banking.csv.1       100%[===================>]   4.66M  --.-KB/s    in 0.02s   

2022-01-20 11:40:33 (243 MB/s) - ‘banking.csv.1’ saved [4882918/4882918]<jupyter_text>資料檢視---1. age (numeric)2. job : type of job (categorical: “admin”, “blue-collar”, “entrepreneur”, “housemaid”, “management”, “retired”, “self-employed”, “services”, “student”, “technician”, “unemployed”, “unknown”)3. marital : marital status (categorical: “divorced”, “married”, “single”, “unknown”)4. education (categorical: “basic.4y”, “basic.6y”, “basic.9y”, “high.school”, “illiterate”, “professional.course”, “university.degree”, “unknown”)5. default: has credit in default? (categorical: “no”, “yes”, “unknown”)6. housing: has housing loan? (categorical: “no”, “yes”, “unknown”)7. loan: has personal loan? (categorical: “no”, “yes”, “unknown”)8. contact: contact communication type (categorical: “cellular”, “telephone”)9. month: last contact month of year (categorical: “jan”, “feb”, “mar”, …, “nov”, “dec”)10. day_of_week: last contact day of the week (categorical: “mon”, “tue”, “wed”, “thu”, “fri”)11. duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y=’no’). The duration is not known before a call is performed, also, after the end of the call, y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model12. campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)13. pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)14. previous: number of contacts performed before this campaign and for this client (numeric)15. poutcome: outcome of the previous marketing campaign (categorical: “failure”, “nonexistent”, “success”)16. emp.var.rate: employment variation rate — (numeric)17. cons.price.idx: consumer price index — (numeric)18. cons.conf.idx: consumer confidence index — (numeric)19. euribor3m: euribor 3 month rate — (numeric)20. nr.employed: number of employees — (numeric)<jupyter_code>import pandas as pd
data = pd.read_csv('banking.csv')
data<jupyter_output><empty_output><jupyter_text>資料處理--- 1. 合併基礎教育程度 2. 將類別型欄位進行One-hot Encoding3. 萃取出X,y4. 切分出training/testing資料集<jupyter_code>import numpy as np
# 合併基礎教育程度 (basic.9y/basic.6y/basic.4y -> Basic)
data['education']=np.where(data['education'] =='basic.9y', 'Basic', data['education'])
data['education']=np.where(data['education'] =='basic.6y', 'Basic', data['education'])
data['education']=np.where(data['education'] =='basic.4y', 'Basic', data['education'])
# 將類別型欄位進行One-hot Encoding
cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']
for var in cat_vars:
    cat_list='var'+'_'+var
    cat_list = pd.get_dummies(data[var], prefix=var)
    data1=data.join(cat_list)
    data=data1
cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']
data_vars=data.columns.values.tolist()
to_keep=[i for i in data_vars if i not in cat_vars]
data_final=data[to_keep]
# 萃取出X,y
x_cols = [col for col in data_final.columns.tolist() if col != 'y']
x_data = data_final[x_cols]
y_data = data_final['y']
X = x_data.values
y = y_data.values
# 切分出training/testing資料集
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)<jupyter_output><empty_output><jupyter_text>建立 Xgboost 模型並進行GridSearchRef: https://www.kaggle.com/tilii7/hyperparameter-grid-search-with-xgboost<jupyter_code># [Step.1.] 建立XGBoostClassifer，objective請使用'binary:logistic'
classifier = None

# [Step.2.] 建立參數搜尋空間 (因課程時間有限，請挑選三個參數進行搜尋，每個參數給兩個選項即可)
# 提示: 用 help(classifier.__init__) 和 help(classifier) 查詢XGBClassifier所用到的參數
param_grid = {?}

# [Step.3.] 建立gridsearch物件 (輸入: classifier, param_grid)
gridsearch = None


# [Step.4.] 執行gridsearch<jupyter_output>Fitting 5 folds for each of 8 candidates, totalling 40 fits
[CV 1/5] END gamma=0.5, min_child_weight=1, n_estimators=100;, score=0.918 total time=   3.5s
[CV 2/5] END gamma=0.5, min_child_weight=1, n_estimators=100;, score=0.916 total time=   3.5s
[CV 3/5] END gamma=0.5, min_child_weight=1, n_estimators=100;, score=0.910 total time=   3.4s
[CV 4/5] END gamma=0.5, min_child_weight=1, n_estimators=100;, score=0.915 total time=   3.4s
[CV 5/5] END gamma=0.5, min_child_weight=1, n_estimators=100;, score=0.913 total time=   3.4s
[CV 1/5] END gamma=0.5, min_child_weight=1, n_estimators=500;, score=0.920 total time=  16.4s
[CV 2/5] END gamma=0.5, min_child_weight=1, n_estimators=500;, score=0.915 total time=  16.5s
[CV 3/5] END gamma=0.5, min_child_weight=1, n_estimators=500;, score=0.911 total time=  16.3s
[CV 4/5] END gamma=0.5, min_child_weight=1, n_estimators=500;, score=0.916 total time=  16.3s
[CV 5/5] END gamma=0.5, min_child_weight=1, n_estimators=500;, score=0.915 total time=  16.4s
[...]<jupyter_text>訓練結果檢視---1. 請給定gridsearch所找到的最佳參數給xgboost進行訓練2. 訓練完畢請對test資料進行預測，並觀察feature importance的合理性。<jupyter_code>print('Start predicting...')
# [Step.5.] 填入最佳參數給xgboost進行建模與預測
xgboost_cls = XGBClassifier(objective = 'binary:logistic', 
                            ?
              )
%time xgboost_fit = xgboost_cls.fit(X_train, y_train)
print('Predicting is over')
# [Step.6.] 成效運算
from  sklearn  import  metrics
def measure_performance(X,y,clf, show_accuracy=True, show_classification_report=True, show_confusion_matrix=True, show_roc_auc = True):
    y_pred = clf.predict(X)
    y_predprob = clf.predict_proba(X)[:,1]
    if show_accuracy:
        print ("Accuracy:{0:.3f}".format(metrics.accuracy_score(y,y_pred))),"\n"

    if show_classification_report:
        print("Classification report")
        print(metrics.classification_report(y,y_pred)),"\n"
        
    if show_confusion_matrix:
        print("Confusion matrix")
        print(metrics.confusion_matrix(y,y_pred)),"\n"  
        
    if show_roc_auc:
        print("ROC AUC Score")
        print(metrics.roc_auc_score(y,y_predprob)),"\n"

XGBoost_grid_measure = measure_performance(X=X_test, y=y_test, clf=xgboost_fit, 
          show_classification_report=True, show_confusion_matrix=True)

# [Step.7.] 建立因子重要性表格，觀察因子重要性是否合理。
importance_table = pd.DataFrame([x_data.columns, xgboost_fit.feature_importances_]).T
importance_table.rename(columns={0: 'column', 1: 'importance'}, inplace=True)
importance_table.sort_values('importance',ascending=False, inplace=True)
importance_table.head(20)<jupyter_output>Start predicting...
CPU times: user 20.6 s, sys: 105 ms, total: 20.7 s
Wall time: 20.6 s
Predicting is over
Accuracy:0.918
Classification report
              precision    recall  f1-score   support

           0       0.94      0.97      0.95      9156
           1       0.66      0.54      0.59      1141

    accuracy                           0.92     10297
   macro avg       0.80      0.75      0.77     10297
weighted avg       0.91      0.92      0.91     10297

Confusion matrix
[[8844  312]
 [ 529  612]]
ROC AUC Score
0.9481265236437345
{'fraction_non_alphanumeric': 0.14139220653878634, 'fraction_numerical': 0.054771383761611774, 'mean_word_length': 4.921428571428572, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '115437', 'license': 'MIT', 'n_tokens_mistral': 3546, 'n_tokens_neox': 3368, 'n_words': 668, 'path': 'gridsearch_student.ipynb', 'repo_name': 'jeffrey82221/colab', 'size': '8298'}