<jupyter_start><jupyter_text>Peer2Peer Loss Given Default PredictionThis solution predicts the Loss due to default of the borrowers who are most likely to default on their Consumer loans in Peer-to-Peer lending. During the training stage, the solution understands the dataset, handles missing data and class imbalance, conducts feature interaction on the training data and selects a subset of best features based on feature importance. It then trains on multiple ML models, identifies the best performing model and tunes it accordingly. This trained model is then selected for prediction on the test data.This sample notebook shows you how to deploy Peer2Peer Loss Given Default Prediction using Amazon SageMaker.> **Note**: This is a reference notebook and it cannot run unless you make changes suggested in the notebook. Pre-requisites:1. **Note**: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.1. Ensure that IAM role used has **AmazonSageMakerFullAccess**1. To deploy this ML model successfully, ensure that:    1. Either your IAM role has these three permissions and you have authority to make AWS Marketplace subscriptions in the AWS account used:         1. **aws-marketplace:ViewSubscriptions**        1. **aws-marketplace:Unsubscribe**        1. **aws-marketplace:Subscribe**      2. or your AWS account has a subscription to Peer-to-Peer Default Loss Prediction. If so, skip step: [Subscribe to the model package](1.-Subscribe-to-the-model-package)    3. Make sure to install below mentioned python libraries before running this notebook     Contents:1. [Subscribe to the model package](1.-Subscribe-to-the-model-package)2. [Usage Instruction](2.-Usage-Instruction)3. [Training](3.-Training) 4. [Sample Data](4.-Sample-Data)5. [Perform batch inference](5.-Perform-batch-inference)6. [Create an endpoint and perform real-time inference](6.-Create-an-endpoint-and-perform-real-time-inference)7. [Clean-up](7.-Clean-up)     Usage instructionsYou can run this notebook one cell at a time (By using Shift+Enter for running a cell).  1. Subscribe to the model package To subscribe to the model package:1. Open the algorithm listing page **Peer2Peer Loss Given Default Prediction**1. On the AWS Marketplace listing, click on the **Continue to subscribe** button.1. On the **Subscribe to this software** page, review and click on **"Accept Offer"** if you and your organization agrees with EULA, pricing, and support terms. 1. Once you click on **Continue to configuration button** and then choose a **region**, you will see a **Product Arn** displayed. This is the model package ARN that you need to specify while creating a deployable model using Boto3. Copy the ARN corresponding to your region and specify the same in the following cell.  2. Usage Instruction The deployed solution has these 2 steps:1: The solution trains on user provided historical training data and builds & saves a ML model which is a representation of the historical data.2: Once the model is generated, the solution can be used to predict on any new data as input file. To achieve this end, the solution deploys the following 2 APIs over AWS Sagemaker:1.Training API: The system trains on user provided historical data with relevant features and builds & saves a ML model.2.Testing API: Once the model is generated, the solution can be used to predict the loss given default for a given test data.  InputSupported content types: `text/csv`* Following is the mandatory inputs for the APIs: * The Target variable should be named**LossGivenDefault**, this features tells us the Percentage of amount lost when Borrower Defaults. Note: 1.	Two separate csv input files are required for training and testing.  * Name of the training file should be train.csv  * Name of the testing file should be sample_input.csv   Output• Content type: ` text/csv`* Solution will generate an output csv file which will contain an additional columns as the prediction along with the columns in the new data.*  The additional columns will be **Label**. Invoking endpoint AWS CLI CommandIf you are using real time inferencing, please create the endpoint first and then use the  following command to invoke it:``` bash !aws sagemaker-runtime invoke-endpoint --endpoint-name $model_name --body fileb://$file_name --content-type 'text/csv' --region us-east-2 sample_output.csv```Substitute the following parameters:* `"model-name"` - name of the inference endpoint where the model is deployed* `file_name` - input csv file name * `text/csv` - content type of the given input * `sample_output.csv` - filename where the inference results are written to.<jupyter_code>import base64 
import uuid
from sagemaker import ModelPackage
import sagemaker as sage
from sagemaker import get_execution_role
from sagemaker import ModelPackage
import boto3
from IPython.display import Image
from PIL import Image
import numpy as np
import pandas as pd
import cv2
from numpy import asarray
role = get_execution_role()

sagemaker_session = sage.Session()

bucket=sagemaker_session.default_bucket()
bucket
# S3 prefixes
common_prefix = "Peer_to_Peer_Default_Loss_Prediction"
training_input_prefix = common_prefix + "/training-input-data"
batch_inference_input_prefix = common_prefix + "/batch-inference-input-data"
sagemaker_session = sage.Session()
# Maintaine this directory structure
TRAINING_WORKDIR = "data/training"

TRAINING_DATA = TRAINING_WORKDIR + "/train.csv"
TRAINING_WORKDIR = "data/training"

training_input = sagemaker_session.upload_data(TRAINING_WORKDIR, key_prefix=training_input_prefix)
print ("Training Data Location " + training_input)<jupyter_output><jupyter_text>3. Training<jupyter_code>import json
import time
from sagemaker.algorithm import AlgorithmEstimator<jupyter_output><empty_output><jupyter_text>Algorithm ARN<jupyter_code>algorithm_arn ='' # put your ARN here
algo = AlgorithmEstimator(
    algorithm_arn=algorithm_arn,
    role=role,
    instance_count=1,
    instance_type='ml.m5.12xlarge',
    base_job_name='Peer-to-Peer-Default-Loss-Prediction')
print ("Now run the training job using algorithm arn %s in region %s" % (algorithm_arn, sagemaker_session.boto_region_name))
algo.fit({'training': training_input})<jupyter_output><jupyter_text>4. Sample Data<jupyter_code># maintaine this directory structure
import os
TRANSFORM_WORKDIR = "data/transform"
sample_data = pd.read_csv(TRANSFORM_WORKDIR + "/sample_input.csv",index_col=0)
sample_data.head(10)<jupyter_output><empty_output><jupyter_text>5. Perform batch inference In this section, you will perform batch inference using multiple input payloads together. If you are not familiar with batch transform, and want to learn more, see these links:1. [How it works](https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-batch-transform.html)2. [How to run a batch transform job](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-batch.html)<jupyter_code>TRANSFORM_WORKDIR = "data/transform"
transform_input = sagemaker_session.upload_data(TRANSFORM_WORKDIR, key_prefix=batch_inference_input_prefix) + "/sample_input.csv"
print("Transform input uploaded to " + transform_input)
transformer = algo.transformer(1, 'ml.m5.12xlarge')
transformer.transform(transform_input, content_type='text/csv')
transformer.wait()

print("Batch Transform output saved to " + transformer.output_path)<jupyter_output><jupyter_text>Inspect the Batch Transform Output in S3<jupyter_code>from urllib.parse import urlparse

parsed_url = urlparse(transformer.output_path)
bucket_name = parsed_url.netloc
file_key = '{}/{}.out'.format(parsed_url.path[1:], 'sample_input.csv')

s3_client = sagemaker_session.boto_session.client('s3')

response = s3_client.get_object(Bucket = sagemaker_session.default_bucket(), Key = file_key)
response_bytes = response['Body'].read().decode('utf-8')
bucketFolder = transformer.output_path.rsplit('/')[3]
import boto3
s3_conn = boto3.client("s3")
bucket_name=sagemaker_session.default_bucket()
with open('sample_output.csv', 'wb') as f:
    s3_conn.download_fileobj(bucket_name, bucketFolder+'/' + 'sample_input.csv' +'.out', f)
    print("Output file loaded from bucket")<jupyter_output>Output file loaded from bucket<jupyter_text>output<jupyter_code>#OUTPUT_WORKDIR = "data/output"

import pandas as pd

out_csv = pd.read_csv("sample_output.csv",index_col=0)
out_csv.head(10)<jupyter_output><empty_output><jupyter_text>6. Create an endpoint and perform real-time inference If you want to understand how real-time inference with Amazon SageMaker works, see [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html).<jupyter_code>model_name='Peer-to-Peer-Default-Loss-Prediction'

content_type='text/csv'

real_time_inference_instance_type='ml.m5.12xlarge'
batch_transform_inference_instance_type='ml.m5.12xlarge'<jupyter_output><empty_output><jupyter_text>Algorithm ARN<jupyter_code>algorithm_arn ='' # put your ARN here<jupyter_output><empty_output><jupyter_text>A. Create an endpoint<jupyter_code>def predict_wrapper(endpoint, session):
    return sage.RealTimePredictor(endpoint, session,content_type)

#create a deployable model from the model package.
model = ModelPackage(role=role,
                    model_package_arn=algorithm_arn,
                    sagemaker_session=sagemaker_session,
                    predictor_cls=predict_wrapper)

#Deploy the model
predictor = algo.deploy(1, 'ml.m5.12xlarge',endpoint_name=model_name)<jupyter_output>..........
-----------!<jupyter_text>Once endpoint has been created, you would be able to perform real-time inference.  B. Create input payload<jupyter_code>import pandas as pd
file_name = 'data/transform/sample_input.csv'
data_frame = pd.read_csv(file_name)<jupyter_output><empty_output><jupyter_text>C. Perform real-time inference<jupyter_code>!aws sagemaker-runtime invoke-endpoint \
    --endpoint-name 'Peer-to-Peer-Default-Loss-Prediction' \
    --body fileb://$file_name \
    --content-type 'text/csv' \
    --region us-east-2 \
    sample_output.csv<jupyter_output>{
    "ContentType": "text/csv; charset=utf-8",
    "InvokedProductionVariant": "AllTraffic"
}<jupyter_text>D. Visualize output<jupyter_code>out_csv = pd.read_csv("sample_output.csv",index_col=0)
out_csv.head(10)<jupyter_output><empty_output><jupyter_text>7. Clean-up  A. Delete the model<jupyter_code>predictor.delete_endpoint()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0894254628745944, 'fraction_numerical': 0.009162053827066235, 'mean_word_length': 5.54528419737664, 'pattern_counts': {'":': 2, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '309394', 'license': 'Apache-2.0', 'n_tokens_mistral': 3111, 'n_tokens_neox': 3001, 'n_words': 1016, 'path': 'Peer2Peer-Loss-Given-Default-Prediction.ipynb', 'repo_name': 'Mphasis-ML-Marketplace/Peer2Peer-Loss-Given-Default-Prediction', 'size': '10478'}