<jupyter_start><jupyter_text>NLU based item search_**Using a pretrained BERT and Elasticsearch KNN to search textually similar items**_------ Contents1. [Background](Background)1. [Setup](Setup)1. [Lauange Translation](Trnslate)1. [SageMaker Model Hosting](Hosting-Model)1. [Build a KNN Index in Elasticsearch](ES-KNN)1. [Evaluate Index Search Results](Searching-with-ES-k-NN)1. [Extensions](Extensions) BackgroundIn this notebook, we'll build the core components of a textually similar items search. Sometime people don't know what exactly they are looking in that case they just type an item descriptions and it will retrive the similar items.One of the core components of textually similar items search is a fixed length sentence/word embedding i,e a  “feature vectors” corrosponds to that text. The reference word/sentence embedding typically are generated offline and must be stored in. So they can be efficiently searched. So generating word/sengtence embedding can be achived by pretrained language model such as BRET(Bidirectional Encoder Representations from Transformers). In our use case we have used pretrained BERT model from sentence-transformers(https://github.com/UKPLab/sentence-transformers).To enable efficient searches for textually similar items, we'll use Amazon SageMaker to generate fixed length sentence embedding i.e “feature vectors” and use KNN algorithim in Amazon Elasticsearch service. KNN for Amazon Elasticsearch Service7.7 lets you search for points in a vector space and find the "nearest neighbors" for those points by cosine similarity (Default is Euclidean distance). Use cases include recommendations (for example, an "other songs you might like" feature in a music application), image recognition, and fraud detection.Here are the steps we'll follow to build textually similar items: After some initial setup, we'll host the pretrained BERT language model in SageMaker PyTorch model server. Then generate feature vectors for Multi-modal Corpus of Fashion Images from *__feidegger__*, a *__zalandoresearch__* dataset. Those feature vectors will be imported in Amazon Elasticsearch KNN Index. Next, we'll explore some sample text queries, and visualize the results.<jupyter_code>#Install tqdm to have progress bar
!pip install tqdm

#install necessary pkg to make connection with elasticsearch domain
!pip install elasticsearch
!pip install requests
!pip install requests-aws4auth
import boto3
import re
import sagemaker
from sagemaker import get_execution_role

role = get_execution_role()

s3_resource = boto3.resource("s3")
s3 = boto3.client('s3')
cfn = boto3.client('cloudformation')

def get_cfn_outputs(stackname):
    outputs = {}
    for output in cfn.describe_stacks(StackName=stackname)['Stacks'][0]['Outputs']:
        outputs[output['OutputKey']] = output['OutputValue']
    return outputs

## Setup variables to use for the rest of the demo
cloudformation_stack_name = "nlu-search"

outputs = get_cfn_outputs(cloudformation_stack_name)

bucket = outputs['s3BucketTraining']
es_host = outputs['esHostName']

outputs<jupyter_output><empty_output><jupyter_text>Downloading Zalando Research dataThe dataset itself consists of 8732 high-resolution images, each depicting a dress from the available on the Zalando shop against a white-background. Each of the images has five textual annotations in German, each of which has been generated by a separate user. **Downloading Zalando Research data**: Data originally from here: https://github.com/zalandoresearch/feidegger  **Citation:**  *@inproceedings{lefakis2018feidegger,*  *title={FEIDEGGER: A Multi-modal Corpus of Fashion Images and Descriptions in German},*  *author={<NAME> and <NAME> and <NAME>},*  *booktitle = {{LREC} 2018, 11th Language Resources and Evaluation Conference},*  *year      = {2018}*  *}*<jupyter_code>## Data Preparation

import os 
import shutil
import json
import tqdm
import urllib.request
from tqdm import notebook
from multiprocessing import cpu_count
from tqdm.contrib.concurrent import process_map

images_path = 'data/feidegger/fashion'
filename = 'metadata.json'

my_bucket = s3_resource.Bucket(bucket)

if not os.path.isdir(images_path):
    os.makedirs(images_path)

def download_metadata(url):
    if not os.path.exists(filename):
        urllib.request.urlretrieve(url, filename)
        
#download metadata.json to local notebook
download_metadata('https://raw.githubusercontent.com/zalandoresearch/feidegger/master/data/FEIDEGGER_release_1.1.json')

def generate_image_list(filename):
    metadata = open(filename,'r')
    data = json.load(metadata)
    url_lst = []
    for i in range(len(data)):
        url_lst.append(data[i]['url'])
    return url_lst


def download_image(url):
    urllib.request.urlretrieve(url, images_path + '/' + url.split("/")[-1])
                    
#generate image list            
url_lst = generate_image_list(filename)     

workers = 2 * cpu_count()

#downloading images to local disk
process_map(download_image, url_lst, max_workers=workers)


# Uploading dataset to S3

files_to_upload = []
dirName = 'data'
for path, subdirs, files in os.walk('./' + dirName):
    path = path.replace("\\","/")
    directory_name = path.replace('./',"")
    for file in files:
        files_to_upload.append({
            "filename": os.path.join(path, file),
            "key": directory_name+'/'+file
        })
        

def upload_to_s3(file):
        my_bucket.upload_file(file['filename'], file['key'])
        
#uploading images to s3
process_map(upload_to_s3, files_to_upload, max_workers=workers)<jupyter_output><empty_output><jupyter_text>Lauange TranslationThis dataset has product descriptions in German, So we will use Amazon Translate for English translation for each German sentence.<jupyter_code>with open(filename) as json_file:
    data = json.load(json_file)
import time
#Define translator function
def translate_txt(data):
    results = {}
    results['filename'] = f's3://{bucket}/data/feidegger/fashion/' + data['url'].split("/")[-1]
    results['descriptions'] = []
    translate = boto3.client(service_name='translate', use_ssl=True)
    for i in data['descriptions']:
        result = translate.translate_text(Text=str(i), 
            SourceLanguageCode="de", TargetLanguageCode="en")
        results['descriptions'].append(result['TranslatedText'])
    time.sleep(5)
    return results
# we are using realtime traslation which will take around ~15 min. 
workers = 8 * cpu_count()

#downloading images to local disk
results = process_map(translate_txt, data, max_workers=workers)
# Saving the translated text in json format in case you need later time
with open('zalando-translated-data.json', 'w', encoding='utf-8') as f:
    json.dump(results, f, ensure_ascii=False, indent=4)<jupyter_output><empty_output><jupyter_text>SageMaker Model HostingIn this section will host the pretrained BERT model into SageMaker Pytorch model server to generte 768 fixed length sentecce embedding from sentence-transformers (https://github.com/UKPLab/sentence-transformers). **Citation:**     @inproceedings{reimers-2019-sentence-bert,    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",    author = "<NAME> and <NAME>",    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",    month = "11",    year = "2019",    publisher = "Association for Computational Linguistics",    url = "http://arxiv.org/abs/1908.10084",}<jupyter_code>!pip install sentence-transformers
#Save the model to disk which we will host at sagemaker
from sentence_transformers import models, SentenceTransformer
saved_model_dir = 'transformer'
if not os.path.isdir(saved_model_dir):
    os.makedirs(saved_model_dir)

model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')
model.save(saved_model_dir)
#Defining defalut bucket for SageMaker pretrained model hosting
sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()
#zip the model .gz format
import tarfile
export_dir = 'transformer'
with tarfile.open('model.tar.gz', mode='w:gz') as archive:
    archive.add(export_dir, recursive=True)
#Upload the model to S3

inputs = sagemaker_session.upload_data(path='model.tar.gz', key_prefix='model')
inputs<jupyter_output><empty_output><jupyter_text>First we need to create a PyTorchModel object. The deploy() method on the model object creates an endpoint which serves prediction requests in real-time. If the instance_type is set to a SageMaker instance type (e.g. ml.m5.large) then the model will be deployed on SageMaker. If the instance_type parameter is set to *__local__* then it will be deployed locally as a Docker container and ready for testing locally.First we need to create a RealTimePredictor class to accept TEXT as input and output JSON. The default behaviour is to accept a numpy array.<jupyter_code>from sagemaker.pytorch import PyTorch, PyTorchModel
from sagemaker.predictor import RealTimePredictor
from sagemaker import get_execution_role

class StringPredictor(RealTimePredictor):
    def __init__(self, endpoint_name, sagemaker_session):
        super(StringPredictor, self).__init__(endpoint_name, sagemaker_session, content_type='text/plain')
           
pytorch_model = PyTorchModel(model_data = inputs, 
                             role=role, 
                             entry_point ='inference.py',
                             source_dir = './code', 
                             framework_version = '1.5.1',
                             predictor_cls=StringPredictor)

predictor = pytorch_model.deploy(instance_type='ml.m5.large', initial_instance_count=3)<jupyter_output><empty_output><jupyter_text>sentence transformers uses BERT pretrained model so it will generate 768 dimension for the given text. we will quickly validate the same in next cell.<jupyter_code># Doing a quick test to make sure model is generating the embeddings
import json
payload = 'a yellow dress that comes about to the knees'
features = predictor.predict(payload)
embedding = json.loads(features)

embedding<jupyter_output><empty_output><jupyter_text>Build a KNN Index in ElasticsearchKNN for Amazon Elasticsearch Service lets you search for points in a vector space and find the "nearest neighbors" for those points by cosine similarity. Use cases include recommendations (for example, an "other songs you might like" feature in a music application), image recognition, and fraud detection.KNN cosine similarity requires Elasticsearch 7.7 or later (Default is Euclidean distance). Full documentation for the Elasticsearch feature, including descriptions of settings and statistics, is available in the Open Distro for Elasticsearch documentation. For background information about the k-nearest neighbors algorithmIn this step we'll get all the translated product descriptions of *__zalandoresearch__* dataset and import those embadings into Elastichseach 7.7 domain.<jupyter_code># setting up the Elasticsearch connection
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
region = 'us-east-1' # e.g. us-east-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

es = Elasticsearch(
    hosts = [{'host': es_host, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)
#KNN index maping
knn_index = {
    "settings": {
        "index.knn": True,
        "index.knn.space_type": "cosinesimil",
        "analysis": {
          "analyzer": {
            "default": {
              "type": "standard",
              "stopwords": "_english_"
            }
          }
        }
    },
    "mappings": {
        "properties": {
           "zalando_nlu_vector": {
                "type": "knn_vector",
                "dimension": 768
            } 
        }
    }
}
#Creating the Elasticsearch index
es.indices.create(index="idx_zalando",body=knn_index,ignore=400)
results[0]['descriptions']
# For each product, we are concatenating all the 
# product descriptions into a single sentence,
# so that we will have one embedding for each product

def concat_desc(results):
    obj = {
        'filename': results['filename'],
    }
    obj['descriptions'] = ' '.join(results['descriptions'])
    return obj

concat_results = map(concat_desc, results)
concat_results = list(concat_results)
concat_results[0]
# defining a function to import the feature vectors corrosponds to each S3 URI into Elasticsearch KNN index
# This process will take around ~10 min.

def es_import(concat_result):
    vector = json.loads(predictor.predict(concat_result['descriptions']))
    es.index(index='idx_zalando',
             body={"zalando_nlu_vector": vector,
                   "image": concat_result['filename'],
                   "description": concat_result['descriptions']}
            )
        
workers = 8 * cpu_count()
    
process_map(es_import, concat_results, max_workers=workers)<jupyter_output><empty_output><jupyter_text>Evaluate Index Search ResultsIn this step we will use SageMaker SDK as well as Boto3 SDK to query the Elasticsearch to retrive the nearest neighbours and retrive the relevent product images from Amazon S3 to display in the notebook.<jupyter_code>#define display_image function
from PIL import Image
import io
def display_image(bucket, key):
    s3_object = my_bucket.Object(key)
    response = s3_object.get()
    file_stream = response['Body']
    img = Image.open(file_stream)
    return display(img)<jupyter_output><empty_output><jupyter_text>SageMaker SDK Method<jupyter_code>#SageMaker SDK approach
import json
payload = 'I want a dress that is yellow and flowery'
features = predictor.predict(payload)
embedding = json.loads(features)
#ES index search
import json
k = 3
idx_name = 'idx_zalando'
res = es.search(request_timeout=30, index=idx_name,
                body={'size': k, 
                      'query': {'knn': {'zalando_nlu_vector': {'vector': embedding, 'k': k}}}})

#Display the image

for x in res['hits']['hits']:
    key = x['_source']['image']
    key = key.replace(f's3://{bucket}/','')
    img = display_image(bucket,key)<jupyter_output><empty_output><jupyter_text>Boto3 Method<jupyter_code>#calling SageMaker Endpoint
client = boto3.client('sagemaker-runtime')
ENDPOINT_NAME = predictor.endpoint
response = client.invoke_endpoint(EndpointName=ENDPOINT_NAME,
                                       ContentType='text/plain',
                                       Body=payload)

response_body = json.loads((response['Body'].read()))

#ES index search
import json
k = 3
idx_name = 'idx_zalando'
res = es.search(request_timeout=30, index=idx_name,
                body={'size': k, 
                      'query': {'knn': {'zalando_nlu_vector': {'vector': response_body, 'k': k}}}})
#Display the image

for i in range(k):
    key = res['hits']['hits'][i]['_source']['image']
    key = key.replace(f's3://{bucket}/','')
    img = display_image(bucket,key)<jupyter_output><empty_output><jupyter_text>Standard Full-Text Amazon ES Search<jupyter_code>fuzzy_search_body = {
    "_source": {
        "excludes": [ "zalando_nlu_vector" ]
    },
    "query": {
        "match" : {
            "description" : {
                "query" : payload
            }
        }
    }
}

result_fuzzy = es.search(request_timeout=30, index=idx_name,
                body=fuzzy_search_body)
    
for x in result_fuzzy['hits']['hits'][:3]:
    key = x['_source']['image']
    key = key.replace(f's3://{bucket}/','')
    img = display_image(bucket,key)<jupyter_output><empty_output><jupyter_text>Deploying a full-stack NLU search application<jupyter_code>s3_resource.Object(bucket, 'backend/template.yaml').upload_file('./backend/template.yaml', ExtraArgs={'ACL':'public-read'})


sam_template_url = f'https://{bucket}.s3.amazonaws.com/backend/template.yaml'

# Generate the CloudFormation Quick Create Link

print("Click the URL below to create the backend API for NLU search:\n")
print((
    'https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review'
    f'?templateURL={sam_template_url}'
    '&stackName=nlu-search-api'
    f'&param_BucketName={outputs["s3BucketTraining"]}'
    f'&param_DomainName={outputs["esDomainName"]}'
    f'&param_ElasticSearchURL={outputs["esHostName"]}'
    f'&param_SagemakerEndpoint={predictor.endpoint}'
))<jupyter_output><empty_output><jupyter_text>Now that you have a working Amazon SageMaker endpoint for extracting image features and a KNN index on Elasticsearch, you are ready to build a real-world full-stack ML-powered web app. The SAM template you just created will deploy an Amazon API Gateway and AWS Lambda function. The Lambda function runs your code in response to HTTP requests that are sent to the API Gateway.<jupyter_code># Review the content of the Lambda function code.
!pygmentize backend/lambda/app.py<jupyter_output><empty_output><jupyter_text>Once the CloudFormation Stack shows CREATE_COMPLETE, proceed to this cell below:<jupyter_code>import json
api_endpoint = get_cfn_outputs('nlu-search-api')['TextSimilarityApi']

with open('./frontend/src/config/config.json', 'w') as outfile:
    json.dump({'apiEndpoint': api_endpoint}, outfile)<jupyter_output><empty_output><jupyter_text>Step 2: Deploy frontend services<jupyter_code># add NPM to the path so we can assemble the web frontend from our notebook code

from os import environ

npm_path = ':/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin'

if npm_path not in environ['PATH']:
    ADD_NPM_PATH = environ['PATH']
    ADD_NPM_PATH = ADD_NPM_PATH + npm_path
else:
    ADD_NPM_PATH = environ['PATH']
    
%set_env PATH=$ADD_NPM_PATH
%cd ./frontend/

!npm install
!npm run-script build
hosting_bucket = f"s3://{outputs['s3BucketHostingBucketName']}"

!aws s3 sync ./build/ $hosting_bucket --acl public-read<jupyter_output><empty_output><jupyter_text>Step 3: Browse your frontend service<jupyter_code>print('Click the URL below:\n')
print(outputs['S3BucketSecureURL'] + '/index.html')<jupyter_output><empty_output><jupyter_text>You should see the following page: ![Website](./frontendExample.png) In the search bar, try typing *__"I want a summery dress that's flowery and yellow"__*. Notice how the results are more relevant in the KNN Search method in contrast with the conventional full-text match search query. The KNN Search method uses features generated from the search text and finds similar product descriptions, which allows it to find similar products, even if the description doesn't have the exact same phrase.  ExtensionsWe have used pretrained BERT model from sentence-transformers. You can fine tune BERT model based on your own usecase with your own data.  CleanupMake sure that you stop the notebook instance, delete the Amazon SageMaker endpoint and delete the Elasticsearch domain to prevent any additional charges.<jupyter_code># Delete the endpoint
predictor.delete_endpoint()

# Empty S3 Contents
training_bucket_resource = s3_resource.Bucket(bucket)
training_bucket_resource.objects.all().delete()

hosting_bucket_resource = s3_resource.Bucket(outputs['s3BucketHostingBucketName'])
hosting_bucket_resource.objects.all().delete()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09639737426991264, 'fraction_numerical': 0.007804827621853518, 'mean_word_length': 4.553386911595867, 'pattern_counts': {'":': 21, '<': 74, '<?xml version=': 0, '>': 74, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '101526', 'license': 'MIT-0', 'n_tokens_mistral': 5813, 'n_tokens_neox': 5599, 'n_words': 1748, 'path': 'nlu-based-item-search.ipynb', 'repo_name': 'OElesin/amazon-sagemaker-nlu-search', 'size': '19389'}