<jupyter_start><jupyter_text>Hyperparameter Tuning on SageMaker<jupyter_code>import os
import boto3
import sagemaker
import pandas as pd
from sagemaker import get_execution_role
from sagemaker.inputs import TrainingInput
from sagemaker.image_uris import retrieve

sess = sagemaker.Session()
role = get_execution_role()
bucket = sess.default_bucket()
training_image = retrieve("linear-learner", boto3.Session().region_name, "1")
print(training_image)
# We retrieve the datasets!
%store -r X_train
%store -r X_test
%store -r X_val
%store -r Y_train
%store -r Y_test
%store -r Y_val<jupyter_output><empty_output><jupyter_text>The very first thing we need to do is get back a reference of the training inputs. If we uploaded to our S3 then we would only get a reference.<jupyter_code>data_dir = "data/"
if not os.path.exists(data_dir):
    os.makedirs(data_dir)

prefix = "boston-dataset"
pd.concat([Y_train, X_train], axis=1).to_csv(
    os.path.join(data_dir, "train.csv"), header=False, index=False
)
pd.concat([Y_val, X_val], axis=1).to_csv(
    os.path.join(data_dir, "validation.csv"), header=False, index=False
)

val_location = sess.upload_data(os.path.join(data_dir, "validation.csv"), key_prefix=prefix)
train_location = sess.upload_data(os.path.join(data_dir, "train.csv"), key_prefix=prefix)
s3_input_train = TrainingInput(s3_data=train_location, content_type="text/csv")
s3_input_validation = TrainingInput(s3_data=val_location, content_type="text/csv")

s3_input_train<jupyter_output><empty_output><jupyter_text>Now we need to do the same as before and configure the training job!<jupyter_code># Create the estimator for the linear learner as before!
# Please use "ml.m5.large" instance type

model = None # FILLME

# Set the hyperparameters we will not tune. Required are mini_batch_size, predictor_type and feature_dim
#FILLME<jupyter_output><empty_output><jupyter_text>And now the beautiful part. In SageMaker you only need configure a HyperparameterTuner class sending the ranges dict. This dict specifies for each tunable parameter what are the ranges, not all algorithms allow you to tune whatever parameter.Finally we define an objective metric and the optimization we want to do. If this were a built-in algorithm, then we would need to specify the metric from a regex on the stdout<jupyter_code>from time import gmtime, strftime
from sagemaker.tuner import (
    IntegerParameter,
    CategoricalParameter,
    ContinuousParameter,
    HyperparameterTuner,
)

# Create a job name that is unique
tuning_job_name = None

# We will tune the learning rate and l1 regularization parameter
hyperparameter_ranges = {
    "learning_rate": ContinuousParameter(0.00001, 1.0),
    "l1": ContinuousParameter(0.00001, 1.0)
}

# Define the objective metric you prefer, you can check all on the old training job you created
objective_metric_name = None

# Specify the HyperparameterTuner class. The params are in https://sagemaker.readthedocs.io/en/stable/api/training/tuner.html
# Please set the max_jobs as 2 and max_parallel_jobs as 1
tuner = HyperparameterTuner(
    # FILLME
)<jupyter_output><empty_output><jupyter_text>Now we are ready. We just fit the tuner and wait for it to run, the result will be many trained models, one per<jupyter_code>tuner.fit(
    {"train": s3_input_train, "validation": s3_input_validation},
    job_name=tuning_job_name,
    include_cls_metadata=False,
)
tuner.wait()   # Key to hook until the end of training<jupyter_output>.................................................!
!<jupyter_text>SageMaker exposes a HyperparameterTuningJobAnalytics class to get back a dataframe with information on the job! Let's analise it.<jupyter_code>tuner_metrics = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)
tuner_metrics.dataframe().sort_values(["FinalObjectiveValue"], ascending=False).head(5)

# You are free to analise the tuner_metrics dataframe!
# Calculate the total training time and amount of jobs that succedeed<jupyter_output>The total training time is 0.03 hours
{'fraction_non_alphanumeric': 0.0928784860557769, 'fraction_numerical': 0.009462151394422311, 'mean_word_length': 5.458199356913183, 'pattern_counts': {'":': 4, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '35201', 'license': 'MIT', 'n_tokens_mistral': 1311, 'n_tokens_neox': 1259, 'n_words': 399, 'path': 'labs/lab4/exercise/perform_HPO.ipynb', 'repo_name': 'axel-sirota/sagemaker-course', 'size': '4016'}