<jupyter_start><jupyter_text>Comprehend Custom Classification and Entity Detection Training & DeploymentThis notebook is used to develop the comprehend custom classification and entity detection training and deployment. The same task/operation can be done using any IDE such as Cloud9 etc. Notebook is used here to give the glimpse of experience to the users about Amazon SageMaker service Get the execution role for the notebook instance. This is the IAM role that you created for your notebook instance. You pass the role to the tuning job<jupyter_code>from sagemaker import get_execution_role
role = get_execution_role()
role<jupyter_output><empty_output><jupyter_text>Initiate the Custom classifcation Training job. This will initiate the training job in comprehend service and produce the trained model.Note: You can get the bucket name from CDK output in the terminal from where you run this deployment.<jupyter_code>import boto3

client = boto3.client('comprehend')

response = client.create_document_classifier(
    DocumentClassifierName='email-classifications-sample',# Enter the name of the classifier
    DataAccessRoleArn=role,
    InputDataConfig={
        'S3Uri': 's3://<Bucket Name from CDK stack resource output>/Comprehend_Training_Data.csv'
    },# This is the public read only bucket having sample data. You can use your bucket for your data.
    LanguageCode='en'
)<jupyter_output><empty_output><jupyter_text>Checking the status of that training. This may take upto 20 mins. Please wait until you get the status "Training Competed"<jupyter_code>import time
modelarn=response["DocumentClassifierArn"]
response_des = client.describe_document_classifier(
    DocumentClassifierArn=modelarn
)
print (response_des['DocumentClassifierProperties']['Status'])
train_status=response_des['DocumentClassifierProperties']['Status']
print ("Training started")
while train_status!='TRAINED':
    print ("Status of the training is :" + train_status)
    time.sleep(30)
    response_des = client.describe_document_classifier(DocumentClassifierArn=modelarn)
    train_status=response_des['DocumentClassifierProperties']['Status']
    
print("Training Completed")<jupyter_output><empty_output><jupyter_text>Creating Endpoint for the Trained model<jupyter_code>#create endpoint for the trained classifer
response_ep = client.create_endpoint(
    EndpointName='email-classifications-endpoint',
    ModelArn=modelarn,
    DesiredInferenceUnits=1,# if you wan to deploy this multiple unit, you can enter more than one. 
    #ClientRequestToken='string',
    Tags=[
        {
            'Key': 'Name',
            'Value': 'email classification'
        },
    ],
    DataAccessRoleArn=role
)<jupyter_output><empty_output><jupyter_text>Check the Endpoint ARN<jupyter_code>response_ep
eparn=response_ep["EndpointArn"]
eparn<jupyter_output><empty_output><jupyter_text>Now test the trined model by sending some sample sentences. Note: Please wait for few mins to deploy this model and endpoint gets created<jupyter_code>response_cd = client.classify_document(
    Text='Can you send the status of the transaction id:278960001',
    EndpointArn=eparn
)
response_cd<jupyter_output><empty_output><jupyter_text>Now this endpoint ARN will be used to classify emails coming from customer via Amazon WorkMail. You will use this ARN for your next CDK deployment  Now lets create the model for Entity Detection. If the above model clasify the incoming email as MONEYTRANSFER, we have to fetch the ststus of the transaction id given in that email. Money Transfer id will prefix with 'MTN' and followed by 7 digit numbers.  Entity Detection model training<jupyter_code>response = client.create_entity_recognizer(
    RecognizerName='email-entity-detection-model',
    DataAccessRoleArn = role,
    InputDataConfig={
        'DataFormat': 'COMPREHEND_CSV',
        'EntityTypes': [
            {
                'Type': 'MTNID'
            },
        ],
        'Documents': {
            'S3Uri': 's3://<Bucket Name from CDK stack resource output>/Comprehend_raw_Data_entity_detection.csv'
        },
        'EntityList': {
            'S3Uri': 's3://<Bucket Name from CDK stack resource output>/sample_entity_data.csv'
        }
    },
    LanguageCode='en'
)
# Check whether training completed or not
entmodelarn=response["EntityRecognizerArn"]
ent_response_des = client.describe_entity_recognizer(
    EntityRecognizerArn=entmodelarn
)
print (ent_response_des['EntityRecognizerProperties']['Status'])
ent_train_status=ent_response_des['EntityRecognizerProperties']['Status']
print ("Training started")
while ent_train_status!='TRAINED':
    print (".")
    time.sleep(30)
    ent_response_des = client.describe_entity_recognizer(EntityRecognizerArn=entmodelarn)
    ent_train_status=ent_response_des['EntityRecognizerProperties']['Status']
    
print("Training Completed")
print(entmodelarn)
# Deploy the model by creating endpoint for this entity detection
ent_response_ep = client.create_endpoint(
    EndpointName='email-entity-detection-endpoint',
    ModelArn=entmodelarn,
    DesiredInferenceUnits=1,# if you wan to deploy this multiple unit, you can enter more than one. 
    #ClientRequestToken='string',
    Tags=[
        {
            'Key': 'Name',
            'Value': 'email entity detection'
        },
    ],
    DataAccessRoleArn=role
)
ent_eparn=ent_response_ep["EndpointArn"]
ent_eparn<jupyter_output><empty_output><jupyter_text>Now test the trined model by sending some sample sentences. Note: Please wait for few mins to deploy this model and endpoint gets created<jupyter_code>ent_response = client.detect_entities(
    Text='Can you send the status of the transaction id: MTN2780001',
    LanguageCode='en',
    EndpointArn=ent_eparn
)
print (ent_response)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.079841351957234, 'fraction_numerical': 0.005863079841351957, 'mean_word_length': 4.95482546201232, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '556848', 'license': 'MIT-0', 'n_tokens_mistral': 1681, 'n_tokens_neox': 1594, 'n_words': 497, 'path': 'notebook/comprehend_class_entity_train_deploy_notebook.ipynb', 'repo_name': 'aws-samples/email-response-automation-comprehend', 'size': '5799'}