<jupyter_start><jupyter_text>Toby Adju<EMAIL>Department of Biosystems and Agricultural EngineeringUniversity of KentuckyFebruary 10, 2022  Random Forest Model deployment in Amazon Sagemaker.In this notebook, I show how I trained and deployed a machine learning model using AWS SageMaker.This notebook was created and run in an Amazon Sagemaker notebook instance. This demonstration is from a project I worked on titled "Machine Learning Approach to Simulating CO2 Fluxes in Cropping Systems". Resources needed1. A dataset: The dataset used in this project was the GRACEnet dataset.Themain purpose of the GRACEnet database is to aggregate information from many studies sothat methods for quantifying GHG emissions and other environmental impacts of croppedand grazed systems can be developed, and to provide scientific evidence for carbon tradingprograms that can help reduce GHG emissions.The original uncleaned and unprocessed data used in this project can be found at: https://data.nal.usda.gov/dataset/gracenet-greenhouse-gas-reduction-through-agricultural-carbon-enhancement-network2. An algorithm: I used the Random Forest algorithm in scikit-learn provided by Amazon SageMaker to train the model using the GRACEnet dataset to predict the CO2 flux in the cropping systems. Resources from Amazon SageMakerA few resources needed for storing your data and running the code in Amazon SageMaker:1. An Amazon Simple Storage Service (Amazon S3) bucket to store the training data and the model artifacts that Amazon SageMaker creates when it trains the model.2. An Amazon SageMaker notebook instance to prepare and process data and to train and deploy a machine learning model.3. A Jupyter notebook to use with the notebook instance to prepare your training data and train and deploy the model.Detailed description of this work is available in our paper: <NAME>., <NAME>. Machine Learning Approach to Simulate Soil CO2 Fluxes under Cropping Systems. Agronomy 2022, 12, 197, doi: https://doi.org/10.3390/agronomy12010197  Fetching the dataset<jupyter_code>import pandas as pd # library for data manipulation and analysis.
import numpy as np # library for scientific computing, provides high-performance, easy to use structures and data analysis tools. 
import matplotlib.pyplot as plt # Plotting library
import seaborn as sns # data visualization library
sns.set(style="darkgrid")

#Uploading a cleaned version of the dataset. To learn more about the data cleaning and manipulation steps for this data, see
# my repository titled "MLSoilCO2Flux"
data = pd.read_csv("New_GHG_Data_Deployment.csv")
data.shape
# Create dataframes for Target and Predictors
 
x_data = pd.DataFrame(np.c_[data ['Air_Temp_DEGC'],data ['Soil_Temp_DEGC'],data ['Soil_Classification'],data ['Crop'],data ['Fert_Ammend_Class']], columns = ['Air_Temp_DEGC','Soil_Temp_DEGC','Soil_Classification','Crop','Fert_Ammend_Class'])
y_data = data ['Carbon_dioxide'] # Create dataset with CO2 flux<jupyter_output><empty_output><jupyter_text>Preparing the data<jupyter_code>from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x_data, y_data, test_size=0.20, random_state=10)                                                    
print("The shape of the X_trainset is",x_train.shape) # The shape of the X_trainset
print("The shape of the y_trainset is", y_train.shape) # The shape of the y_trainset

print("The shape of the X_testset is", x_test.shape) # The shape of the X_testset
print("The shape of the y_testset is", y_test.shape) # The shape of the y_testset
x_train.head()
#Get variable/feature names
feature_names=list(x_data.columns.values)
feature_names
trainX = pd.DataFrame(x_train, columns=feature_names)# Asssign feature names to the training data set
trainX['target'] = y_train

testX = pd.DataFrame(x_test, columns=feature_names)# Asssign feature names to the testing data set
testX['target'] = y_test
trainX.head()
trainX.to_csv('GHG_train.csv')# Save the training dataset
testX.to_csv('GHG_test.csv')# Save the testing dataset<jupyter_output><empty_output><jupyter_text>Data IngestionThe training and testing files are then copied to S3 for Amazon SageMaker's managed training to pickup.This should be within the same region as the Notebook Instance, training, and hosting.<jupyter_code>import datetime
import tarfile

import boto3 # AWS SDK for python. Provides low-level access to AWS services
from sagemaker import get_execution_role
import sagemaker

m_boto3 = boto3.client('sagemaker') 

sess = sagemaker.Session()

region = sess.boto_session.region_name

bucket = 'bucket-name'  #  Bucket is a logical unit of storage in AWS S3

print('Using bucket ' + bucket)
# send data to S3. SageMaker will take training data from s3
trainpath = sess.upload_data(
    path='GHG_train.csv', bucket=bucket,
    key_prefix='sagemaker/sklearncontainer')

testpath = sess.upload_data(
    path='GHG_test.csv', bucket=bucket,
    key_prefix='sagemaker/sklearncontainer')<jupyter_output><empty_output><jupyter_text>Prepare a Scikit-learn Training ScriptHere, we write the scitkit-learn training script that will be used to train the random forest model.The training script is similar to a training script you might run outside of SageMaker, but you can access useful properties about the training environment through various environment variables. For example:- SM_MODEL_DIR:  A string that represents the path where the training job writes the model artifacts to. After training, artifacts in this directory are uploaded to S3 for model hosting.- SM_OUTPUT_DATA_DIR: A string representing the filesystem path to write output artifacts to. Output artifacts may include checkpoints, graphs, and other files to save, not including model artifacts. These artifacts are compressed and uploaded to S3 to the same S3 prefix as the model artifacts.Supposing two input channels, ‘train’ and ‘test’, were used in the call to the Scikit-learn estimator’s fit() method, the following will be set, following the format “SMCHANNEL[channel_name]”:- SM_CHANNEL_TRAIN: A string representing the path to the directory containing data in the ‘train’ channel- SM_CHANNEL_TEST: Same as above, but for the ‘test’ channel.<jupyter_code>%%writefile script.py

import argparse
import os
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
import joblib
from sklearn.metrics import explained_variance_score, r2_score



# inference functions ---------------
def model_fn(model_dir):
    clf = joblib.load(os.path.join(model_dir, "model.joblib"))
    return clf

if __name__ =='__main__':

    print('extracting arguments')
    parser = argparse.ArgumentParser()

    # hyperparameters sent by the client are passed as command-line arguments to the script.
    parser.add_argument('--n-estimators', type=int, default=100)
    parser.add_argument('--max_leaf_nodes', type=int, default=10)
    

#Fit model to training set
    

    # Data, model, and output directories
   
    parser.add_argument('--model-dir', type=str, default=os.environ.get('SM_MODEL_DIR'))
    parser.add_argument('--train', type=str, default=os.environ.get('SM_CHANNEL_TRAIN'))
    parser.add_argument('--test', type=str, default=os.environ.get('SM_CHANNEL_TEST'))
    parser.add_argument('--train-file', type=str, default='GHG_train.csv')
    parser.add_argument('--test-file', type=str, default='GHG_test.csv')
    
    
    args, _ = parser.parse_known_args()
    
    print('reading data')
    train_df = pd.read_csv(os.path.join(args.train, args.train_file))
    test_df = pd.read_csv(os.path.join(args.test, args.test_file))


    
    print('building training and testing datasets')
    attributes = ['Air_Temp_DEGC', 'Soil_Temp_DEGC', 'Soil_Classification',
       'Crop', 'Fert_Ammend_Class']
    X_train = train_df[attributes]
    X_test = test_df[attributes]
    y_train = train_df['target']
    y_test = test_df['target']
    
    # train
    print('training model')
    model = RandomForestRegressor(
        n_estimators=args.n_estimators,
        max_leaf_nodes =args.max_leaf_nodes,
        n_jobs=-1)
    
    model.fit(X_train, y_train)
     
    # persist model
    path = os.path.join(args.model_dir, "model.joblib")
    joblib.dump(model, path)
    print('model persisted at ' + path)
    
    # print explained_variance_score 
    print('validating model')
    predictions = model.predict(X_test)
    print("Explained Variance Score is " + str(explained_variance_score(y_test, predictions).round(2)))
    print("R2 score : %.2f" % r2_score(y_test,predictions))<jupyter_output>Overwriting script.py<jupyter_text>Local TrainingScript arguments allows us to remove from the script any SageMaker-specific configuration, and run locally<jupyter_code>! python script.py --n-estimators 100 \
                   --max-leaf-nodes 10 \
                   --model-dir ./ \
                   --train ./ \
                   --test ./ \<jupyter_output>extracting arguments
reading data
building training and testing datasets
training model
model persisted at ./model.joblib
validating model
Explained Variance Score is 0.72
R2 score : 0.72<jupyter_text>SageMaker Training - Launching a training job with the Python SDK Create an EstimatorYou run Scikit-learn training scripts on SageMaker by creating SKLearn Estimators. Call the fit method on a SKLearn Estimator to start a SageMaker training job. The following code sample shows how you train a custom Scikit-learn script named “script.py”, passing in two hyperparameters ('n-estimators', 'max_leaf_nodes'), and using two input channel directories (‘train’ and ‘test’).<jupyter_code>#We use the Estimator from the SageMaker Python SDK
from sagemaker.sklearn.estimator import SKLearn
sklearn_estimator = SKLearn(
    entry_point='script.py',
    role = get_execution_role(),
    train_instance_count=1,
    train_instance_type='ml.m4.xlarge',
    framework_version='0.23-1',
    base_job_name='rf-scikit',
    hyperparameters = {'n-estimators': 500,
                       'max_leaf_nodes': 16 
                       })
# launch training job, with asynchronous call
sklearn_estimator.fit({'train':trainpath, 'test': testpath}, wait=False)<jupyter_output><empty_output><jupyter_text>Deploy to a real-time endpoint Deploy with Python SDKAn Estimator could be deployed directly after training, with an Estimator.deploy() but here we are using the more extensive process of creating a model from s3 artifacts, that could be used to deploy a model that was trained in a different session or even out of SageMaker. After the estimator finishes training, we deploy it to a SageMaker endpoint:<jupyter_code>sklearn_estimator.latest_training_job.wait(logs='None')
artifact = m_boto3.describe_training_job(
    TrainingJobName=sklearn_estimator.latest_training_job.name)['ModelArtifacts']['S3ModelArtifacts']

print('Model artifact persisted at ' + artifact)

predictor = sklearn_estimator.deploy(instance_type='ml.m4.xlarge',initial_instance_count=1)<jupyter_output>------!<jupyter_text>Invoke with the Python SDK<jupyter_code>print(predictor.predict(testX[feature_names]))
predictions = predictor.predict(testX[feature_names])# Making predictions with the trained model
from sklearn.metrics import r2_score
print("R2 score : %.2f" % r2_score(testX['target'],predictions))<jupyter_output>R2 score : 0.77<jupyter_text>As you can see from the evaluation metrics, the trained model attained an R2 of 0.77. This score can be improved with furthurhyperparameter tuning, however I chose to use two hyperparameters ('n-estimators', 'max_leaf_nodes). It is possible to implement automatic hyperparameter tuning using "GridSearch" or "RandomSearch" packages provided in scikit learn.  Deleting the endpoint !<jupyter_code>sagemaker.Session().delete_endpoint(predictor.endpoint)
# bucket_to_delete = boto3.resource('s3').Bucket(bucket_name)
# bucket_to_delete.objects.all().delete()<jupyter_output>The endpoint attribute has been renamed in sagemaker>=2.
See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.<jupyter_text>If you want to come back to this notebook after you deployed the SageMaker endpoint, you can use the following snippet of code to invoke it:<jupyter_code># sklearn_predictor = Predictor(endpoint_name='randomforestregressor-endpoint',
#                               sagemaker_session=sess,
#                               serializer=NumpySerializer(),
#                               deserializer=NumpyDeserializer())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0811242414564037, 'fraction_numerical': 0.008783136378153944, 'mean_word_length': 4.745412844036697, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 40, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '48161', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 3766, 'n_tokens_neox': 3592, 'n_words': 1239, 'path': 'Random Forest Model Deployment.ipynb', 'repo_name': 'Toby4321/Random-Forest-Model-deployment-in-Amazon-Sagemaker', 'size': '12548'}