<jupyter_start><jupyter_text>Oge: Image Classification using the Afrifashion1600 DatasetThis notebook lists all the steps that you need to complete the complete this project. You will need to complete all the TODOs in this notebook as well as in the README and the two python scripts included with the starter code.**TODO**: Give a helpful introduction to what this notebook is for. Remember that comments, explanations and good documentation make your project informative and professional.**Note:** This notebook has a bunch of code and markdown cells with TODOs that you have to complete. These are meant to be helpful guidelines for you to finish your project while meeting the requirements in the project rubrics. Feel free to change the order of these the TODO's and use more than one TODO code cell to do all your tasks.<jupyter_code>!pip install smdebug --ignore-installed
import os
from dataclasses import dataclass
from IPython.display import HTML
from time import gmtime
from time import strftime

import boto3
import sagemaker
from sagemaker import get_execution_role
from sagemaker.debugger import DebuggerHookConfig
from sagemaker.debugger import FrameworkProfile
from sagemaker.debugger import ProfilerConfig
from sagemaker.debugger import ProfilerRule
from sagemaker.debugger import Rule
from sagemaker.debugger import rule_configs
from sagemaker.model_monitor import DataCaptureConfig
from sagemaker.pytorch import PyTorch
from sagemaker.serializers import IdentitySerializer
from sagemaker.tuner import CategoricalParameter
from sagemaker.tuner import ContinuousParameter
from sagemaker.tuner import HyperparameterTuner
from sagemaker.tuner import IntegerParameter
from smdebug.trials import create_trial
from smdebug.profiler.analysis.notebook_utils.training_job import TrainingJob
from smdebug.profiler.analysis.notebook_utils.timeline_charts import TimelineCharts

from oge.utils import plot_tensor
sess = sagemaker.Session()<jupyter_output><empty_output><jupyter_text>`Configuration`<jupyter_code>@dataclass
class Config:
    BUCKET = sess.default_bucket()
    REGION = sess.boto_region_name
    EXECUTION_ROLE = sagemaker.get_execution_role()
    SEED = 23
    INSTANCE_COUNT = 1
    INSTANCE_TYPE = "ml.p2.xlarge"
    MAX_JOBS = 4
    MAX_PARALLEL_JOBS = 1
    MONITORING_INTERVAL_MILLIS = 100
    PROFILE_EVERY_X_STEPS = 10
config = Config()
os.environ["DEFAULT_S3_BUCKET"] = f"s3://{config.BUCKET}/"<jupyter_output><empty_output><jupyter_text>`Dataset`TODO: Explain what dataset you are using for this project. Maybe even give a small overview of the classes, class distributions etc that can help anyone not familiar with the dataset get a better understand of it.<jupyter_code># Download and unzip dataset
!mkdir data
!wget https://github.com/DataScienceNigeria/Research-Papers-by-Data-Science-Nigeria/raw/master/AFRIFASHION1600:%20A%20Contemporary%20African%20Fashion%20Dataset%20for%20Computer%20Vision/data/AFRIFASHION1600.zip -P data
!unzip data/AFRIFASHION1600.zip -d data && rm data/AFRIFASHION1600.zip
# Split dataset into train and validation sets
!python oge/dataset.py --dataset_path data/AFRIFASHION1600 --train_size 0.7 #--seed {c.SEED}
# Upload dataset to S3
!mkdir -p logs
!aws s3 sync data/AFRIFASHION1600 ${DEFAULT_S3_BUCKET}afrifashion1600/ > logs/data_upload.log 2>&1
# Confirm dataswet upload to S3
!aws s3 ls ${DEFAULT_S3_BUCKET}afrifashion1600/<jupyter_output>PRE train/
                           PRE val/<jupyter_text>`Hyperparameter Tuning`**TODO:** This is the part where you will finetune a pretrained model with hyperparameter tuning. Remember that you have to tune a minimum of two hyperparameters. However you are encouraged to tune more. You are also encouraged to explain why you chose to tune those particular hyperparameters and the ranges.**Note:** You will need to use the `hpo.py` script to perform hyperparameter tuning.<jupyter_code># TODO: Declare your HP ranges, metrics etc.
hyperparameter_ranges = {
    "train_batch_size": CategoricalParameter([32, 64, 128, 256, 512]),
    "lr": ContinuousParameter(0.001, 0.1),
    "n_epochs": CategoricalParameter([3, 5, 7]),
    "normalize": CategoricalParameter([True, False])
}
objective_metric_name = "average test loss"
objective_type = "Minimize"
metric_definitions = [{
    "Name": "average test loss",
    "Regex": "Test set: Average loss: ([0-9\\.]+)"
}]
estimator = PyTorch(
    source_dir = "oge",
    entry_point="train_model.py",
    base_job_name="oge-afrifashion-resnet50-pytorch",
    role=config.EXECUTION_ROLE,
    instance_count=config.INSTANCE_COUNT,
    instance_type=config.INSTANCE_TYPE,   # TODO
    framework_version="1.8",
    py_version="py36"
)
tuner = HyperparameterTuner(
    estimator,
    objective_metric_name,
    hyperparameter_ranges,
    metric_definitions,
    max_jobs=config.MAX_JOBS,
    max_parallel_jobs=config.MAX_PARALLEL_JOBS,   
    objective_type=objective_type
)
# TODO: Fit your HP Tuner
tuner.fit(
    {
        "train": "s3://sagemaker-us-east-1-661396466295/afrifashion1600/train",
        "test": "s3://sagemaker-us-east-1-661396466295/afrifashion1600/val"
    }
)
# TODO: Get the best estimators and the best HPs

best_estimator = tuner.best_estimator() #TODO

#Get the hyperparameters of the best trained model
best_hyperparameters = best_estimator.hyperparameters()
print(best_hyperparameters)<jupyter_output>2021-11-27 00:49:38 Starting - Preparing the instances for training
2021-11-27 00:49:38 Downloading - Downloading input data
2021-11-27 00:49:38 Training - Training image download completed. Training in progress.
2021-11-27 00:49:38 Uploading - Uploading generated training model
2021-11-27 00:49:38 Completed - Training job completed
{'_tuning_objective_metric': '"average test loss"', 'lr': '0.0015630256742041576', 'n_epochs': '"5"', 'normalize': '"True"', 'sagemaker_container_log_level': '20', 'sagemaker_estimator_class_name': '"PyTorch"', 'sagemaker_estimator_module': '"sagemaker.pytorch.estimator"', 'sagemaker_job_name': '"oge-afrifashion-resnet50-pytorch-2021-11-27-00-14-36-639"', 'sagemaker_program': '"train_model.py"', 'sagemaker_region': '"us-east-1"', 'sagemaker_submit_directory': '"s3://sagemaker-us-east-1-661396466295/oge-afrifashion-resnet50-pytorch-2021-11-27-00-14-36-639/source/sourcedir.tar.gz"', 'train_batch_size': '"32"'}<jupyter_text>Model Profiling and DebuggingTODO: Using the best hyperparameters, create and finetune a new model**Note:** You will need to use the `train_model.py` script to perform model profiling and debugging.<jupyter_code>rules = [
    Rule.sagemaker(rule_configs.loss_not_decreasing()),
    Rule.sagemaker(rule_configs.overfit()),
    Rule.sagemaker(rule_configs.overtraining()),
    Rule.sagemaker(rule_configs.poor_weight_initialization()),
    ProfilerRule.sagemaker(rule_configs.LowGPUUtilization()),
    ProfilerRule.sagemaker(rule_configs.BatchSize()),
]
debugger_config = DebuggerHookConfig(
    hook_parameters={"train.save_interval": "50", "eval.save_interval": "10"}
)
# TODO: Set up debugging and profiling rules and hooks
profiler_config = ProfilerConfig(
    system_monitor_interval_millis=config.MONITORING_INTERVAL_MILLIS,
    framework_profile_params=FrameworkProfile(num_steps=config.PROFILE_EVERY_X_STEPS)
)
# These parameters are from the Hyperparameter Tuning section above
hyperparameters = {
    "train_batch_size": 32,
    "n_epochs": 5,
    "lr": 0.0015630256742041576,
    "normalize": True
}
# TODO: Create and fit an estimator
estimator = PyTorch(
    source_dir="oge",
    entry_point="train_model.py",
    base_job_name="oge-afrifashion-resnet50-pytorch-debug-and-profile",
    role=get_execution_role(),
    instance_count=config.INSTANCE_COUNT,
    instance_type=config.INSTANCE_TYPE,   # TODO
    framework_version="1.8",
    py_version="py36",
    hyperparameters=hyperparameters,
    profiler_config=profiler_config,
    debugger_config=debugger_config,
    rules=rules
)
estimator.fit(
    {
        "train": "s3://sagemaker-us-east-1-661396466295/afrifashion1600/train",
        "test": "s3://sagemaker-us-east-1-661396466295/afrifashion1600/val"
    }
)<jupyter_output>2021-12-01 21:19:48 Starting - Starting the training job...
2021-12-01 21:20:05 Starting - Launching requested ML instancesLossNotDecreasing: InProgress
Overfit: InProgress
Overtraining: InProgress
PoorWeightInitialization: InProgress
LowGPUUtilization: InProgress
ProfilerReport: InProgress
.........
2021-12-01 21:21:39 Starting - Preparing the instances for training.........
2021-12-01 21:23:20 Downloading - Downloading input data......
2021-12-01 21:24:20 Training - Downloading the training image..................LowGPUUtilization: IssuesFound
ProfilerReport: InProgress
........[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device[0m
[34mbash: no job control in this shell[0m
[34m2021-12-01 21:28:29,243 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training[0m
[34m2021-12-01 21:28:29,266 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.[0m
[34m2021-12-01 21:28:29,276 sagemake[...]<jupyter_text>System Utilization<jupyter_code>training_job_name = estimator.latest_training_job.name
print(f"Training jobname: {training_job_name}")
print(f"Region: {config.REGION}")
client = estimator.sagemaker_session.sagemaker_client
description = client.describe_training_job(TrainingJobName=estimator.latest_training_job.name)
print(description)
training_job = TrainingJob(training_job_name, config.REGION)
training_job.wait_for_sys_profiling_data_to_be_available()
# TODO: Plot a debugging output.
system_metrics_reader = training_job.get_systems_metrics_reader()
system_metrics_reader.refresh_event_file_list()

view_timeline_charts = TimelineCharts(
    system_metrics_reader,
    framework_metrics_reader=None,
    select_dimensions=["GPU", "CPU"]
)<jupyter_output>[2021-12-01 21:43:28.552 datascience-1-0-ml-t3-medium-1abf3407f667f989be9d86559395:19 INFO metrics_reader_base.py:134] Getting 7 event files
select events:['.*']
select dimensions:['GPU', 'CPU']
filtered_events:{'total', 'cpu0', 'cpu3', 'MemoryUsedPercent', 'ReadThroughputInBytesPerSecond', 'cpu2', 'ReceiveBytesPerSecond', 'gpu0', 'TransmitBytesPerSecond', 'IOPS', 'WriteThroughputInBytesPerSecond', 'cpu1'}
filtered_dimensions:{'GPUMemoryUtilization-nodeid:algo-1', 'CPUUtilization-nodeid:algo-1', 'GPUUtilization-nodeid:algo-1'}<jupyter_text>Check Training Performance<jupyter_code>trial = create_trial(estimator.latest_job_debugger_artifacts_path())
trial.tensor_names()
plot_tensor(trial, "nll_loss_output_0")<jupyter_output><empty_output><jupyter_text>**TODO**: Is there some anomalous behaviour in your debugging output? If so, what is the error and how will you fix it?  **TODO**: If not, suppose there was an error. What would that error look like and how would you have fixed it?  Profiler Report<jupyter_code># TODO: Display the profiler output
rule_output_path = estimator.output_path + estimator.latest_training_job.job_name + "/rule-output"
print(rule_output_path)
!aws s3 ls {rule_output_path} --recursive
!aws s3 cp {rule_output_path} ./ --recursive
profiler_report_name = [
    rule["RuleConfigurationName"]
    for rule in estimator.latest_training_job.rule_job_summary()
    if "Profiler" in rule["RuleConfigurationName"]
][0]
profiler_report = HTML(filename=profiler_report_name + "/profiler-output/profiler-report.html")

# Write profiler report to file
with open('html_file.html', 'w') as f:
    f.write(profiler_report.data)

# Display Profiler report
profiler_report<jupyter_output><empty_output><jupyter_text>Model Deployment<jupyter_code>endpoint_name = "oge-afrifashion1600-endpoint-" + strftime("%Y-%m-%d-%H-%M-%S", gmtime())
print(endpoint_name)
data_capture_config = DataCaptureConfig(
    enable_capture=True,
    sampling_percentage=100,
    destination_s3_uri=f"s3://{config.BUCKET}/data_capture"
)
predictor = estimator.deploy(
    instance_type="ml.p2.xlarge",
    initial_instance_count=1,
    endpoint_name=endpoint_name,
    data_capture_config=data_capture_config
)
serializer = IdentitySerializer("image/png")
predictor.serializer = serializer
with open("./data/AFRIFASHION1600/val/buba_and_trouser/Buba and Trouser_183.png", "rb") as f:
    payload = f.read()

response = predictor.predict(data=payload)
print(response)
# TODO: Remember to shutdown/delete your endpoint once your work is done
predictor.delete_endpoint()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10244721689059501, 'fraction_numerical': 0.045665387076135634, 'mean_word_length': 5.8296013107591484, 'pattern_counts': {'":': 16, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '94937', 'license': 'MIT', 'n_tokens_mistral': 4458, 'n_tokens_neox': 4061, 'n_words': 956, 'path': 'train_and_deploy.ipynb', 'repo_name': 'theyorubayesian/oge', 'size': '12504'}