<jupyter_start><jupyter_text>Retail Demo Store - Search WorkshopWelcome to the Retail Demo Store Search Workshop. In this module we'll be configuring the Retail Demo Store Search service to allow searching for product data via Elasticsearch. An Elasticsearch domain should already be provisioned for you in your AWS environment.Recommended Time: 20 Minutes  SetupTo get started, we need to perform a bit of setup. Walk through each of the following steps to configure your environment to interact with the Amazon Personalize Service.  Import Dependencies and Setup Boto3 Python ClientsThrougout this workshop we will need access to some common libraries and clients for connecting to AWS services.<jupyter_code># Import Dependencies

import boto3
import json
import pandas as pd
import time
import requests
import csv

from random import randint
from botocore.exceptions import ClientError

# Setup Clients

servicediscovery = boto3.client('servicediscovery')
ssm = boto3.client('ssm')
es_service = boto3.client('es')<jupyter_output><empty_output><jupyter_text>Create Index and Bulk Index Product Data  Get Products Service InstanceWe will be creating a new Elasticsearch Index and indexing our product data so that our users can search for products. To do this, first we will be pulling our Product data from [Products Service](https://github.com/aws-samples/retail-demo-store/tree/master/src/products) that is deployed as part of the Retail Demo Store. To connect to the Products Service we will use Service Discovery to discover an instance of the Products Service, and then connect directly to that service instances to access our data.<jupyter_code>response = servicediscovery.discover_instances(
    NamespaceName='retaildemostore.local',
    ServiceName='products',
    MaxResults=1,
    HealthStatus='HEALTHY'
)

assert len(response['Instances']) > 0, 'Products service instance not found; check ECS to ensure it launched cleanly'

products_service_instance = response['Instances'][0]['Attributes']['AWS_INSTANCE_IPV4']
print('Service Instance IP: {}'.format(products_service_instance))<jupyter_output><empty_output><jupyter_text>Download and Explore the Products DatasetNow that we have the IP address of one of our Products Service instances, we can connect to it and download our product catalog. To more easily explore our data, we will convert the json response form the Products Service into a dataframe and print it as a table.<jupyter_code>response = requests.get('http://{}/products/all'.format(products_service_instance))
products = response.json()
products_df = pd.DataFrame(products)
pd.set_option('display.max_rows', 5)

products_df<jupyter_output><empty_output><jupyter_text>Install Elasticsearch Python LibraryWe will use the Python Elasticsearch library to connect to our Amazon Elasticsearch cluster, create a new index, and then bulk index our product data. First, we need to install the Elasticsearch library into our environment. We will install the latest version of the Python library that still had [support for open source distributions](https://aws.amazon.com/blogs/opensource/keeping-clients-of-opensearch-and-elasticsearch-compatible-with-open-source/), such as Amazon Elasticsearch.<jupyter_code>!pip install --upgrade pip
!pip install --upgrade elasticsearch==7.13.4<jupyter_output><empty_output><jupyter_text>Find Elasticsearch Domain EndpointBefore we can configure the Elasticsearch client, we need to determine the endpoint for the Elasticsearch domain created in your AWS environment. We will accomplish this by looking for the Elasticsearch domain with tag key of `Name` and tag value of `retaildemostore`. This tag was associated with the Amazon Elasticsearch domain that was created when the project was deployed to your AWS account.<jupyter_code>elasticsearch_domain_endpoint = None

domains_response = es_service.list_domain_names()

for domain_name in domains_response['DomainNames']:
    describe_response = es_service.describe_elasticsearch_domain(
        DomainName=domain_name['DomainName']
    )
    
    tags_response = es_service.list_tags(ARN=describe_response['DomainStatus']['ARN'])

    domain_match = False
    for tag in tags_response['TagList']:
        if tag['Key'] == 'Name' and tag['Value'] == 'retaildemostore':
            domain_match = True
            break
            
    if domain_match:
        elasticsearch_domain_endpoint = describe_response['DomainStatus']['Endpoints']['vpc']
        break;

print('Elasticsearch domain endpoint: ' + str(elasticsearch_domain_endpoint))

assert elasticsearch_domain_endpoint, 'Elasticsearch domain endpoint could not be determined. Ensure Elasticsearch domain has been successfully created and has "retaildemostore" tag before continuing.'<jupyter_output><empty_output><jupyter_text>Configure and Create Elasticsearch Client<jupyter_code>from elasticsearch import Elasticsearch

ES_HOST = {
    'host' : elasticsearch_domain_endpoint,
    'port' : 443,
    'scheme' : 'https',
}
INDEX_NAME = 'products'
TYPE_NAME = 'product'
ID_FIELD = 'id'

es = Elasticsearch(hosts = [ES_HOST])<jupyter_output><empty_output><jupyter_text>Prepare Product Data for IndexingBatch products into chunks to avoid timeouts.<jupyter_code>bulk_datas = [] 
bulk_data = []

bulk_datas.append(bulk_data)

max_data_len = 100

for product in products:
    data_dict = product

    op_dict = {
        "index": {
            "_index": INDEX_NAME, 
            "_type": TYPE_NAME, 
            "_id": data_dict[ID_FIELD]
        }
    }
    bulk_data.append(op_dict)
    bulk_data.append(data_dict)
    
    if len(bulk_data) >= max_data_len:
        bulk_data = []
        bulk_datas.append(bulk_data)<jupyter_output><empty_output><jupyter_text>Check for and Delete Existing IndexesIf the products index already exists, we'll delete it so everything gets rebuilt from scratch.<jupyter_code>if es.indices.exists(INDEX_NAME):
    print("Deleting '%s' index..." % (INDEX_NAME))
    res = es.indices.delete(index = INDEX_NAME)
    print(" response: '%s'" % (res))
else:
    print('Index does not exist. Nothing to do.')<jupyter_output><empty_output><jupyter_text>Create Index<jupyter_code>request_body = {
    "settings" : {
        "number_of_shards": 1,
        "number_of_replicas": 0
    }
}
print("Creating '%s' index..." % (INDEX_NAME))
res = es.indices.create(index = INDEX_NAME, body = request_body)
print(" response: '%s'" % (res))<jupyter_output><empty_output><jupyter_text>Perform Bulk Indexing<jupyter_code>print("Bulk indexing...")
for bulk_data in bulk_datas:
    res = es.bulk(index = INDEX_NAME, body = bulk_data, refresh = True)
    
print("Done")<jupyter_output><empty_output><jupyter_text>Validate Results Through ElasticsearchTo verify that the products have been successfully indexed, let's perform a wildcard search for `brush*` directly against the Elasticsearch index.<jupyter_code>res = es.search(index = INDEX_NAME, body={"query": {"wildcard": { "name": "brush*"}}})
print(json.dumps(res, indent=2))<jupyter_output><empty_output><jupyter_text>Validate Results Through Search ServiceFinally, let's verify that the Retail Demo Store's [Search service](https://github.com/aws-samples/retail-demo-store/tree/master/src/search) can successfully query from the Elasticsearch index as well. Discover Search ServiceFirst we need to get the address to the [Search service](https://github.com/aws-samples/retail-demo-store/tree/master/src/search).<jupyter_code>response = servicediscovery.discover_instances(
    NamespaceName='retaildemostore.local',
    ServiceName='search',
    MaxResults=1,
    HealthStatus='HEALTHY'
)

assert len(response['Instances']) > 0, 'Search service instance not found; check ECS to ensure it launched cleanly'

search_service_instance = response['Instances'][0]['Attributes']['AWS_INSTANCE_IPV4']
print('Service Instance IP: {}'.format(search_service_instance))<jupyter_output><empty_output><jupyter_text>Call Search ServiceLet's call the service's index page which simply echos the service name.<jupyter_code>!curl {search_service_instance}<jupyter_output><empty_output><jupyter_text>Finally, let's do the same `brush` search through the Search service. We should get back the same item IDs as the direct Elasticsearch query above.<jupyter_code>!curl {search_service_instance}/search/products?searchTerm='brush'<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08804204993429698, 'fraction_numerical': 0.0034643411778760005, 'mean_word_length': 5.530421216848674, 'pattern_counts': {'":': 9, '<': 57, '<?xml version=': 0, '>': 60, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '132915', 'license': 'MIT-0', 'n_tokens_mistral': 2416, 'n_tokens_neox': 2372, 'n_words': 757, 'path': 'workshop/0-StartHere/Search.ipynb', 'repo_name': 'xtrim-serverless/retail-demo-store', 'size': '8371'}