<jupyter_start><jupyter_text>Tuning Your Hyper-Parameters We will use SageMaker tuning to automate the searching process effectively. Specifically, we specify a range, or a list of possible values in the case of categorical hyperparameters, for each of the hyperparameter that we plan to tune. SageMaker hyperparameter tuning will automatically launch multiple training jobs with different hyperparameter settings, evaluate results of those training jobs based on a predefined "objective metric", and select the hyperparameter settings for future attempts based on previous results. For each hyperparameter tuning job, we will give it a budget (max number of training jobs) and it will complete once that many training jobs have been executed.In this example, we are using SageMaker Python SDK to set up and manage the hyperparameter tuning job. We first configure the training jobs the hyperparameter tuning job will launch by initiating an estimator, which includes:* The container image for the algorithm (XGBoost)* Configuration for the output of the training jobs* The values of static algorithm hyperparameters, those that are not specified will be given default values* The type and number of instances to use for the training jobs<jupyter_code>!pip install -q boto3
!pip install -q xgboost==0.90
import boto3
import sagemaker
import pandas as pd

sess   = sagemaker.Session()
bucket = sess.default_bucket()
role = sagemaker.get_execution_role()
region = boto3.Session().region_name

sm = boto3.Session().client(service_name='sagemaker', region_name=region)
%store -r spark_processing_job_s3_output_prefix
print('Previous Spark Processing Job Name: {}'.format(spark_processing_job_s3_output_prefix))<jupyter_output><empty_output><jupyter_text>Specify the S3 Location of the Features<jupyter_code>prefix_train = '{}/output/tfidf-labeled-split-balanced-noheader-train'.format(spark_processing_job_s3_output_prefix)
prefix_validation = '{}/output/tfidf-labeled-split-balanced-noheader-validation'.format(spark_processing_job_s3_output_prefix)
prefix_test = '{}/output/tfidf-labeled-split-balanced-noheader-test'.format(spark_processing_job_s3_output_prefix)

balanced_tfidf_without_header_train_path = './{}'.format(prefix_train)
balanced_tfidf_without_header_validation_path = './{}'.format(prefix_validation)
balanced_tfidf_without_header_test_path = './{}'.format(prefix_test)

balanced_tfidf_without_header_train_s3_uri = 's3://{}/{}'.format(bucket, prefix_train)
balanced_tfidf_without_header_validation_s3_uri = 's3://{}/{}'.format(bucket, prefix_validation)
balanced_tfidf_without_header_test_s3_uri = 's3://{}/{}'.format(bucket, prefix_test)

s3_input_train_data = sagemaker.s3_input(s3_data=balanced_tfidf_without_header_train_s3_uri, content_type='text/csv')
s3_input_validation_data = sagemaker.s3_input(s3_data=balanced_tfidf_without_header_validation_s3_uri, content_type='text/csv')
s3_input_test_data = sagemaker.s3_input(s3_data=balanced_tfidf_without_header_test_s3_uri, content_type='text/csv')

print(s3_input_train_data.config)
print(s3_input_validation_data.config)
print(s3_input_test_data.config)
from sagemaker.amazon.amazon_estimator import get_image_uri 
import json

builtin_container_uri = get_image_uri(region_name=region,                                
                                      repo_name='xgboost', 
                                      repo_version='0.90-2')
print(builtin_container_uri)

model_output_path = 's3://{}/models/tuner/'.format(bucket)
print(model_output_path)

xgb_estimator = sagemaker.estimator.Estimator(image_name=builtin_container_uri, 
                                              role=role, 
                                              train_instance_count=2,
                                              train_instance_type='ml.c5.4xlarge',
                                              output_path=model_output_path, 
                                              sagemaker_session=sess,
                                              # enable_cloudwatch_metrics=True
                                             )

xgb_estimator.set_hyperparameters(objective='binary:logistic',
                                  num_round=1,
                                  max_depth=5)<jupyter_output><empty_output><jupyter_text>Setup Hyper-Parameter Ranges to Explore<jupyter_code>from sagemaker.tuner import IntegerParameter
from sagemaker.tuner import ContinuousParameter
from sagemaker.tuner import HyperparameterTuner

hyperparameter_ranges = {
    'alpha': ContinuousParameter(0, 1000, scaling_type="Auto"),
    'colsample_bylevel': ContinuousParameter(0.1, 1,scaling_type="Logarithmic"),
    'colsample_bytree': ContinuousParameter(0.5, 1, scaling_type='Logarithmic'),
    'eta': ContinuousParameter(0.1, 0.5, scaling_type='Logarithmic'),
    'gamma':ContinuousParameter(0, 5, scaling_type='Auto'),
    'lambda': ContinuousParameter(0,100,scaling_type='Auto'),
    'max_delta_step': IntegerParameter(0,10,scaling_type='Auto'),
    'max_depth': IntegerParameter(0,10,scaling_type='Auto'),
    'min_child_weight': ContinuousParameter(0,10,scaling_type='Auto'),
    'num_round': IntegerParameter(1,4000,scaling_type='Auto'),
    'subsample': ContinuousParameter(0.5,1,scaling_type='Logarithmic')}

objective_metric_name = 'validation:auc'

tuner = HyperparameterTuner(
    xgb_estimator,
    objective_metric_name,
    hyperparameter_ranges,
    max_jobs=1,
    max_parallel_jobs=1,
    strategy='Bayesian'
)<jupyter_output><empty_output><jupyter_text>Start the Hyper-Parameter Tuning Experiment<jupyter_code>tuner.fit({'train': s3_input_train_data, 
           'validation': s3_input_validation_data}, 
           include_cls_metadata=False)
from pprint import pprint

job_description = sm.describe_hyper_parameter_tuning_job(
    HyperParameterTuningJobName=tuner.latest_tuning_job.job_name
)

status = job_description['HyperParameterTuningJobStatus']

print('\n')
print(status)
print('\n')
pprint(job_description)

if status != 'Completed':
    job_count = tuning_job_result['TrainingJobStatusCounters']['Completed']
    print('Not yet complete, but {} jobs have completed.')
    
    if job_description.get('BestTrainingJob', None):
        print("Best candidate:")
        pprint(tuning_job_result['BestTrainingJob']['TrainingJobName'])
        pprint(tuning_job_result['BestTrainingJob']['FinalHyperParameterTuningJobObjectiveMetric'])
    else:
        print("No training jobs have reported results yet.")<jupyter_output><empty_output><jupyter_text>Compare Candidates with SageMaker ExperimentsModel tuning automatically creates a new experiment and tracks the results of each job within the tuning experiment.<jupyter_code>from sagemaker.analytics import HyperparameterTuningJobAnalytics

exp = HyperparameterTuningJobAnalytics(
    sagemaker_session=sess, 
    hyperparameter_tuning_job_name=tuner.latest_tuning_job.name
)
df = exp.dataframe()

df.sort_values('FinalObjectiveValue', ascending=0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08771428571428572, 'fraction_numerical': 0.012428571428571428, 'mean_word_length': 4.614274258219727, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '154076', 'license': 'Apache-2.0', 'n_tokens_mistral': 2132, 'n_tokens_neox': 2016, 'n_words': 419, 'path': '08_optimize/archive/99_Tune_Hyper_Parameters.ipynb', 'repo_name': 'MarcusFra/workshop', 'size': '7000'}