<jupyter_start><jupyter_text>[Module 6.2] Batch Transform InferenceBatch Transform은 추론을 1 건씩 하는 것이 아니라, 1회 추론시에 그룹(배치, 벌크) 단위로 합니다.아래 예제는 334건 (334명의 유저)에 대해서 배치 추론을 하여 예측 합니다.이 노트북에서는 아래와 같은 내용을 진행 합니다.- Bactch Transform Job 실행- Test 데이터의 결과 생성 (Churn의 유무를 True, False로 구분)- Ground Truth 데이터와 비교 (Confustion Matrix)- Evaluation Metrics 확인 (F1, Precision, Recall, Accuracy)실행 시간은 약 5분 소요 됩니다.<jupyter_code>%store -r
import boto3

from sagemaker.transformer import Transformer
import pandas as pd
import sagemaker

from time import strftime, gmtime

sess = sagemaker.Session()
role = sagemaker.get_execution_role()
client = boto3.client('sagemaker')

timestamp_prefix = strftime("%Y-%m-%d-%H-%M-%S", gmtime())
transform_job_name = 'churn-batch-transform-' + timestamp_prefix

output_prefix = 'output'
output_path = 's3://{}/{}/{}'.format(bucket, prefix, output_prefix)<jupyter_output><empty_output><jupyter_text>Batch Transfomer 생성model_name은 이전의 노트북에서 저장한 Inference Pipeline 임<jupyter_code># Make Transformer object
transformer = Transformer(base_transform_job_name = transform_job_name,
                          model_name = inference_pipeline_model_name,
                          instance_count=1,
                          instance_type='ml.m4.xlarge',
                          output_path= output_path
                         )<jupyter_output><empty_output><jupyter_text>Test할 데이타 준비<jupyter_code>import os

s3_input_test_url = s3_input_test.config['DataSource']['S3DataSource']['S3Uri']
s3_input_test_url = os.path.join(s3_input_test_url, 'batch_transform_test.csv')
print(s3_input_test_url)
test_df = pd.read_csv(s3_input_test_url, header=None)
print("Test Data Shape: ", test_df.shape)
test_df.head()<jupyter_output>Test Data Shape:  (334, 20)<jupyter_text>Batch Transformer 실행  아래는 약 8분 정도가 소요 됩니다. 아래 셀의 [*] 의 표시가 [숫자] (에: [6])로 바뀔 때까지 기다려 주세요<jupyter_code>%%time 

transformer.transform(s3_input_test_url,
                      content_type='text/csv'
                     )

transformer.wait()<jupyter_output>[36mArguments: serve[0m
[36m[2020-08-27 09:51:44 +0000] [1] [INFO] Starting gunicorn 19.7.1[0m
[36m[2020-08-27 09:51:44 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)[0m
[36m[2020-08-27 09:51:44 +0000] [1] [INFO] Using worker: gevent[0m
[36m[2020-08-27 09:51:44 +0000] [38] [INFO] Booting worker with pid: 38[0m
[36m[2020-08-27 09:51:45 +0000] [39] [INFO] Booting worker with pid: 39[0m
[36m[2020-08-27:09:51:45:INFO] Model loaded successfully for worker : 38[0m
[36m[2020-08-27 09:51:45 +0000] [40] [INFO] Booting worker with pid: 40[0m
[36m[2020-08-27:09:51:45:INFO] Model loaded successfully for worker : 39[0m
[36m[2020-08-27 09:51:45 +0000] [41] [INFO] Booting worker with pid: 41[0m
[36m[2020-08-27:09:51:45:INFO] Model loaded successfully for worker : 40[0m
[36m[2020-08-27:09:51:45:INFO] Model loaded successfully for worker : 41[0m
[34m2020/08/27 09:51:44 [crit] 17#17: *1 connect() to unix:/tmp/gunicorn.sock failed (2: No such file or directory) while con[...]<jupyter_text>예측 결과 확인결과 파일을 로컬 노트북으로 복사<jupyter_code>import os
os.makedirs('output', exist_ok=True)

! aws s3 cp {output_path}/batch_transform_test.csv.out 'output/batch_transform_test_output.csv'<jupyter_output>download: s3://sagemaker-ap-northeast-2-057716757052/sagemaker/customer-churn/output/batch_transform_test.csv.out to output/batch_transform_test_output.csv<jupyter_text>**예측 결과를 확인 (334 명 중에서 22명이 이탈 예상. 약 7%)**<jupyter_code>df_churn = pd.read_csv("output/batch_transform_test_output.csv", names=["Churn"])
df_churn.Churn.value_counts()<jupyter_output><empty_output><jupyter_text>실제 데이타와 예측 결과 비교**실제 데이타는 334명 중에서 48명이 이탈을 했음. 약 14%**<jupyter_code>df_true = pd.read_csv("churn_data/batch_transform_test_true.csv", header=None)
df_true_label = df_true.iloc[:,-1]
# Replace "False." --> False, "True." --> True
def clean_raw_data(churn_raw):
    df_churn = churn_raw.copy()
    df_churn = df_churn.replace({'True.' : True, 'False.' : False})
    
    return df_churn

df_true = clean_raw_data(df_true_label)
print("Ground Truth Vales:")
df_true.value_counts()<jupyter_output>Ground Truth Vales:<jupyter_text>Confusion Matrix 생성y_actual: 실제로 이탈을 했는지 안했는지의 값 y_predict: 에측값으로서 이탈을 했는지 안했는지의 값<jupyter_code>data = { 'y_actual' : df_churn.Churn.to_list(),
         'y_predict' : df_true.to_list()
       }
df_churn_comp = pd.DataFrame(data, columns=['y_actual','y_predict' ])
df_churn_comp.head(10)
import matplotlib.pyplot as plt
import seaborn as sn
import pandas as pd

def plot_conf_mat(cm, classes, title, cmap = plt.cm.Greens):
    print(cm)
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
        horizontalalignment="center",
        color="black" if cm[i, j] > thresh else "black")

        plt.tight_layout()
        plt.ylabel('Predicted label')
        plt.xlabel('Actual label')
                
import itertools
import numpy as np
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format='retina'

cm = confusion_matrix(y_true= df_churn_comp['y_actual'], 
                      y_pred= df_churn_comp['y_predict'])

plt.figure()
fig, ax = plt.subplots(figsize=(7,4))
plot_conf_mat(cm, 
              classes=['Churn_No','Churn_Yes'], 
              title='Confusion Matrix')
plt.show()<jupyter_output>[[283  31]
 [  3  17]]<jupyter_text>Evaluation Metrics 확인<jupyter_code>from sklearn.metrics import f1_score, precision_score, recall_score, accuracy_score
print("f1 metric: ", f1_score(y_true = df_churn_comp['y_actual'],y_pred = df_churn_comp['y_predict'] ))
print("precision : ", precision_score(y_true = df_churn_comp['y_actual'],y_pred = df_churn_comp['y_predict'] ))
print("recall_score : ", recall_score(y_true = df_churn_comp['y_actual'],y_pred = df_churn_comp['y_predict'] ))
print("accuracy: ", accuracy_score(y_true = df_churn_comp['y_actual'],y_pred = df_churn_comp['y_predict'] ))<jupyter_output>f1 metric:  0.5
precision :  0.3541666666666667
recall_score :  0.85
accuracy:  0.8982035928143712<jupyter_text>[참고] 차원 축소의 값의 실험아래는 차원 축소의 값을 69 --> n 개로 실험 했을때의 값입니다.f1 score 기준으로 n=20 일때가 59.4 로 가장 좋았습니다.XGBoost의 하이퍼 파라미터의 값은 고정 하여 실험을 했기에,더 나은 성능을 위해서는 XGBoost의 하이퍼 파라미터의 튜닝이 필요 합니다.<jupyter_code># n=5
# f1 metric:  0.19354838709677416
# precision :  0.125
# recall_score :  0.42857142857142855
# accuracy:  0.8502994011976048

# n = 10
# f1 metric:  0.3548387096774194
# precision :  0.22916666666666666
# recall_score :  0.7857142857142857
# accuracy:  0.8802395209580839

# n = 15
# f1 metric:  0.5753424657534247
# precision :  0.4375
# recall_score :  0.84
# accuracy:  0.907185628742515

# n = 20
# f1 metric:  0.5945945945945945
# precision :  0.4583333333333333
# recall_score :  0.8461538461538461
# accuracy:  0.9101796407185628

# n = 25

# f1 metric:  0.5633802816901409
# precision :  0.4166666666666667
# recall_score :  0.8695652173913043
# accuracy:  0.907185628742515

# n= 30
# f1 metric:  0.5633802816901409
# precision :  0.4166666666666667
# recall_score :  0.8695652173913043
# accuracy:  0.907185628742515

# n=40
# f1 metric:  0.47058823529411764
# precision :  0.3333333333333333
# recall_score :  0.8
# accuracy:  0.8922155688622755<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12959534514678656, 'fraction_numerical': 0.1128008463369479, 'mean_word_length': 4.631422189128816, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '99067', 'license': 'MIT', 'n_tokens_mistral': 3803, 'n_tokens_neox': 3491, 'n_words': 608, 'path': '6.2.inference_batch_transform.ipynb', 'repo_name': 'gonsoomoon-ml/churn-prediction-workshop2', 'size': '7562'}