<jupyter_start><jupyter_text>Part 1: Packaging your Algorithm Amazon SageMaker allows you to package your own algorithms and train and host it on Sagemaker. Here we package a scikit-learn implementation of decision trees for use with SageMaker.  Contents of the container* **Dockerfile** describes how to build your Docker container image. * **build_and_push.sh** script uses the Dockerfile to build your container images and then pushes it to ECR.* **decision_trees** contains the files that will be installed in the container.  The files in the container are:* **nginx.conf**,the configuration file for the nginx front-end.* **predictor.py**,program implements the Flask web server and the decision tree predictions for this app.* **serve** program starts when the container is started for hosting, launches the gunicorn server which runs multiple instances of the Flask app defined in predictor.py. * **train**, program that is invoked when the container is run for training.* SageMaker will look to run an executable program named "train" for training and "serve" for hosting.* Or you can specify any ENTRYPOINT in your Dockerfile which has train() and serve() functions defined within.* **wsgi.py**,a small wrapper used to invoke the Flask app.<jupyter_code>!cat container/Dockerfile<jupyter_output><empty_output><jupyter_text>Building and registering the container* Build the container image using docker build * Push the container image to ECR using docker push. * Get the region defined in the current configuration (default to us-west-2 if none defined)* Looks for an ECR repository in the current account and current default region. If the repository doesn't exist, the script will create it.* Get the login command from ECR and execute it directly* Build the docker image locally with the image name * Push it to ECR with the full name.* On a SageMaker Notebook Instance, the docker daemon may need to be restarted in order to detect your network configuration correctly.(This is a known issue.)<jupyter_code>%%sh

algorithm_name=decision-trees-sample

cd container

chmod +x decision_trees/train
chmod +x decision_trees/serve

account=$(aws sts get-caller-identity --query Account --output text)

region=$(aws configure get region)
region=${region:-us-west-2}

fullname="${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest"

aws ecr describe-repositories --repository-names "${algorithm_name}" > /dev/null 2>&1

if [ $? -ne 0 ]
then
    aws ecr create-repository --repository-name "${algorithm_name}" > /dev/null
fi

$(aws ecr get-login --region ${region} --no-include-email)

if [ -d "/home/ec2-user/SageMaker" ]; then
  sudo service docker restart
fi

docker build  -t ${algorithm_name} .
docker tag ${algorithm_name} ${fullname}

docker push ${fullname}<jupyter_output><empty_output><jupyter_text>Part 2: Training and Hosting the AlgorithmOnce you have your container packaged, you can use it to train and serve models.   Import packages * os -  provides a portable way of using operating system dependent functionality.* gmtime - Convert a time expressed in seconds since the epoch to a struct_time in UTC.* strftime - Convert a tuple or struct_time representing a time as returned by gmtime()<jupyter_code>import os
from time import gmtime, strftime<jupyter_output><empty_output><jupyter_text>Importing some standard python packages * csv- module provides objects to read and write sequences.* itertools - module implements a number of iterator building blocks.* numpy  - package for scientific computing with Python.* pandas - library providing data structures and data analysis tools for Python.<jupyter_code>import csv
import itertools
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Importing amazon packages* boto3 - The AWS SDK for Python to write software that uses Amazon services like S3 and EC2.* psycopg2 - popular PostgreSQL database adapter for the Python* sagemaker - Python SDK for training and deploying machine learning models on Amazon SageMaker.* get_execution_role - Return the role ARN whose credentials are used to call the API.* csv_serializer - Defines csv as the behavior for serialization of input data.<jupyter_code>import boto3
import psycopg2
import sagemaker as sage
from sagemaker.predictor import csv_serializer
from sagemaker import get_execution_role
con=psycopg2.connect(dbname= 'loonydb1', host='myloony-db.cwwbfulhovv2.us-east-2.redshift.amazonaws.com', 
port= '5439', user= 'masteruser', password= '<PASSWORD>')
cur = con.cursor()
query="select * from public.irisdata ;"
cur.execute(query)
results = cur.fetchall()
fp = open('iris.csv','w')
c= csv.writer(fp, lineterminator='\n') 
for row in results:
    print (row)
    c.writerow(row)
fp.close()<jupyter_output><empty_output><jupyter_text>Upload the data for training* Set the bucket path* Create a sagemaker session* Create a bucket and upload the training data.<jupyter_code>prefix = 'scikit-byoc'
sess = sage.Session()
data_location = sess.upload_data('iris.csv', key_prefix=prefix)<jupyter_output><empty_output><jupyter_text>Hosting the model * Get the account and region information* Get the conatiner image* Get the IAM role credentials* Instantiate an estimator* Invoke the fit method to train the model* Deploy the model<jupyter_code>account = sess.boto_session.client('sts').get_caller_identity()['Account']
account
region = sess.boto_session.region_name
region
image = '{}.dkr.ecr.{}.amazonaws.com/decision-trees-sample'.format(account, region)
role = get_execution_role()
role
tree = sage.estimator.Estimator(image,
                       role, 1, 'ml.c4.2xlarge',
                       output_path="s3://{}/output".format(sess.default_bucket()),
                       sagemaker_session=sess)
tree.fit(data_location)
predictor = tree.deploy(1, 'ml.m4.xlarge', serializer=csv_serializer)<jupyter_output><empty_output><jupyter_text>Validate the model* extract some of the data we used for training* pass in the data to the predictor object<jupyter_code>shape=pd.read_csv("iris.csv", header=None)
df = shape[50:110]
names = df[0].values.T.tolist()
names
test_X =df.drop(df.columns[0], axis=1) 
test_X
results = predictor.predict(test_X.values).decode('utf-8')
results=results.split()
print(results)
print (np.array(names) == np.array(results))<jupyter_output><empty_output><jupyter_text>Delete endpoint<jupyter_code>sess.delete_endpoint(predictor.endpoint)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09050054238338757, 'fraction_numerical': 0.00573376724004339, 'mean_word_length': 5.3648915187376724, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '46531', 'license': 'Apache-2.0', 'n_tokens_mistral': 1904, 'n_tokens_neox': 1828, 'n_words': 700, 'path': 'Bring_your_own_container_scikit.ipynb', 'repo_name': 'Katniss394/AWS-Sagemaker-Samples', 'size': '6454'}