<jupyter_start><jupyter_text>Model 1: XGBoost initial model<jupyter_code>bayes_xgb = xgb.XGBClassifier(base_score=0.5, 
                              booster='gbtree', 
                              colsample_bylevel=1,
                              colsample_bytree=0.7, 
                              eval_metric='auc', 
                              gamma=1.0,
                              learning_rate=0.015, 
                              max_delta_step=0, 
                              max_depth=7,
                              min_child_weight=1, 
                              missing=None, # I have imputed missing values
                              n_estimators=236, 
                              n_jobs=-1,
                              nthread=-1, 
                              objective='binary:logistic', 
                              random_state=0,
                              reg_alpha=1e-09, 
                              reg_lambda=1, 
                              scale_pos_weight=1, 
                              seed=None,
                              silent=1, 
                              subsample=0.5)
%%time
bayes_xgb.fit(train, train_labels)
def xgb_important_features(model, feature_names,top=10,width=10,height=6):
    
    f_imp = pd.DataFrame(model.feature_importances_*100,index=feature_names)
    f_imp.columns = ['Importance']
    f_imp = f_imp.sort_values('Importance',ascending=False)
    
    f_imp_top = f_imp[:top]
    f_imp_top = f_imp_top.sort_values('Importance',ascending=True)
    
    plt.figure(figsize=(width, height))
    ax = plt.subplot( )
 
    
    ax.barh(y =f_imp_top.index, width =f_imp_top['Importance'])
    
    return(f_imp)


importance = xgb_important_features(bayes_xgb, train.columns,top =12, width=10, height=5 )
importance<jupyter_output><empty_output><jupyter_text>Predict test<jupyter_code>test_preds = bayes_xgb.predict_proba(test) 
# predict_proba - predict the probability of each data example being of a given class.

# Probability of being influences by marketing campaign.
# Most of examples will have small probability
fig, axs = plt.subplots(1,2,figsize=(15,3))
axs[0].hist(test_preds[:,1],bins = 50, log = False)
axs[1].hist(test_preds[:,1],bins = 50, log = True)
plt.show()
test_preds = pd.DataFrame({'LNR':test.index ,'RESPONSE':  test_preds[:,1] })
test_preds.to_csv("../submissions/xgboost_init.csv",sep=",", index=False)
test_preds<jupyter_output><empty_output><jupyter_text>Model 1 test score: `0.77172` --- Hyperparameter tuningInitial "guessed" model is already doing better than linear learner (on `Kaggle`). Here I will  Train/validation splitI am going to split my train dataset in order to address the target metric `validation:auc` of our xgboost model.<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(train, train_labels, train_size=0.925)
plt.hist(Y_train)
print(np.unique(Y_test,return_counts=True))
plt.hist(Y_test)<jupyter_output>(array([0., 1.]), array([3185,   38]))<jupyter_text>Upload to S3<jupyter_code># Test - without true labels (which we don't have!)
import os
data_dir = "../arvato_data_processed/xgboost"

test.to_csv( os.path.join(data_dir+ "test.csv"), header=False, index=False)
# Train/validation
pd.concat([Y_train, X_train], axis=1).to_csv(  os.path.join(data_dir+ "train.csv"), header=False, index=False)
pd.concat([Y_test, X_test], axis=1).to_csv(  os.path.join(data_dir+ "validation.csv"), header=False, index=False)


prefix = 'arvato-xgboost-tuning'

test_location = sagemaker_session.upload_data(os.path.join(data_dir, 'test.csv'), key_prefix=prefix)
val_location = sagemaker_session.upload_data(os.path.join(data_dir, 'validation.csv'), key_prefix=prefix)
train_location = sagemaker_session.upload_data(os.path.join(data_dir, 'train.csv'), key_prefix=prefix)<jupyter_output><empty_output><jupyter_text>Define an initial model<jupyter_code># As stated above, we use this utility method to construct the image name for the training container.
container = get_image_uri(sagemaker_session.boto_region_name, 'xgboost')

# Now that we know which container to use, we can construct the estimator object.
xgb = sagemaker.estimator.Estimator(container, # The name of the training container
                                    role,      # The IAM role to use (our current role in this case)
                                    train_instance_count=1, # The number of instances to use for training
                                    train_instance_type='ml.m4.xlarge', # The type of instance ot use for training
                                    output_path='s3://{}/{}/output'.format(sagemaker_session.default_bucket(), prefix),
                                                                        # Where to save the output (the model artifacts)
                                    sagemaker_session=sagemaker_session) # The current SageMaker session<jupyter_output>'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.
WARNING:root:There is a more up to date SageMaker XGBoost image. To use the newer image, please set 'repo_version'='1.0-1'. For example:
	get_image_uri(region, 'xgboost', '1.0-1').
WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.<jupyter_text>Hyperparameter tuning<jupyter_code>xgb.set_hyperparameters(base_score=0.5,  # global bias
                              booster='gbtree', 
                              colsample_bylevel=1, # Subsample ratio of columns for each level.
                              colsample_bytree=0.9, # Subsample ratio of columns when constructing each tree.
                              eval_metric='auc', 
                              gamma=1.0,
                              eta=0.015, 
                              max_delta_step=0, 
                              max_depth=7,
                              min_child_weight=1, 
                              missing=None, # I have imputed missing values
                              n_estimators=250, 
                              n_jobs=-1,
                              objective='binary:logistic', 
                              random_state=0,
                              reg_alpha=1e-09, 
                              reg_lambda=1e-5, 
                              scale_pos_weight=1, 
                              seed=123,
                              silent=1, 
                              subsample=0.95,
                              num_round=250, # max number of rounds 
                         early_stopping_rounds=10, #  Activates early stopping. Validation metric needs to improve at least once in every early_stopping_rounds round(s) to continue training
)
from sagemaker.tuner import IntegerParameter, ContinuousParameter, HyperparameterTuner

xgb_hyperparameter_tuner = HyperparameterTuner(estimator = xgb, # The estimator object to use as the basis for the training jobs.
                                               objective_metric_name = 'validation:auc', # The metric used to compare trained models.
                                               objective_type = 'Maximize', # Whether we wish to minimize or maximize the metric.
                                               max_jobs = 50, # The total number of models to train
                                               max_parallel_jobs = 10, # The number of models to train in parallel
                                               hyperparameter_ranges = {
                                                    'max_depth': IntegerParameter(3, 12),
                                                    'eta'      : ContinuousParameter(0.01, 0.5),
                                                    'min_child_weight': IntegerParameter(2, 8),
                                                    'subsample': ContinuousParameter(0.5, 0.9),
                                                    'gamma': ContinuousParameter(0, 10),
                                                    'colsample_bytree': ContinuousParameter(0.5, 1),
                                                   'alpha' : ContinuousParameter(1e-09, 1e-1),
                                                   'lambda' : ContinuousParameter(1e-09, 1e-1)
                                               })

# This is a wrapper around the location of our train and validation data, to make sure that SageMaker
# knows our data is in csv format.
s3_input_train = sagemaker.s3_input(s3_data=train_location, content_type='csv')
s3_input_validation = sagemaker.s3_input(s3_data=val_location, content_type='csv')

xgb_hyperparameter_tuner.fit({'train': s3_input_train, 'validation': s3_input_validation})
%%time
xgb_hyperparameter_tuner.wait()<jupyter_output>................................................................................................................................................................................................................................................!
CPU times: user 1.48 s, sys: 97.2 ms, total: 1.58 s
Wall time: 20min 15s<jupyter_text>Select the best model<jupyter_code>xgb_hyperparameter_tuner.best_training_job()
xgb_attached = sagemaker.estimator.Estimator.attach(xgb_hyperparameter_tuner.best_training_job())<jupyter_output>WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.<jupyter_text>Predict the test<jupyter_code>xgb_transformer = xgb_attached.transformer(instance_count = 1, instance_type = 'ml.m4.xlarge')
xgb_transformer.transform(test_location, content_type='text/csv', split_type='Line')
xgb_transformer.wait()
!aws s3 cp --recursive $xgb_transformer.output_path $data_dir
!cat $data_dir/test.csv.out | wc -l
test_preds = pd.read_csv(os.path.join(data_dir,"test.csv.out"),header=None)
test_preds.columns =[ 'RESPONSE']
test_preds['LNR'] = test.index.values
test_preds = test_preds[['LNR','RESPONSE']]

test_preds.to_csv("../submissions/xgboost_optimized.csv",sep=",", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1124789040007942, 'fraction_numerical': 0.018365928720341505, 'mean_word_length': 1.92, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '371225', 'license': 'MIT', 'n_tokens_mistral': 2940, 'n_tokens_neox': 2781, 'n_words': 606, 'path': 'Supervised/02_XGBoost.ipynb', 'repo_name': 'vaxherra/Arvato-Customer-Segmentation', 'size': '10073'}