<jupyter_start><jupyter_text>IntroductionWord2Vec is a popular algorithm used for generating dense vector representations of words in large corpora using unsupervised learning. The resulting vectors have been shown to capture semantic relationships between the corresponding words and are used extensively for many downstream natural language processing (NLP) tasks like sentiment analysis, named entity recognition and machine translation.   SageMaker BlazingText which provides efficient implementations of Word2Vec on- single CPU instance- single instance with multiple GPUs - P2 or P3 instances- multiple CPU instances (Distributed training) In this notebook, we demonstrate how BlazingText can be used for distributed training of word2vec using multiple CPU instances.  SetupLet's start by specifying:- The S3 bucket and prefix that you want to use for training and model data. This should be within the same region as the Notebook Instance, training, and hosting. If you don't specify a bucket, SageMaker SDK will create a default bucket following a pre-defined naming convention in the same region. - The IAM role ARN used to give SageMaker access to your data. It can be fetched using the **get_execution_role** method from sagemaker python SDK.<jupyter_code>import sagemaker
from sagemaker import get_execution_role
import boto3
import json

sess = sagemaker.Session()

role = get_execution_role()
print(role) # This is the role that SageMaker would use to leverage AWS resources (S3, CloudWatch) on your behalf

bucket = sess.default_bucket() # Replace with your own bucket name if needed
print(bucket)
prefix = 'sagemaker/DEMO-blazingtext-text8' #Replace with the prefix under which you want to store the data if needed<jupyter_output><empty_output><jupyter_text>Data IngestionNext, we download a dataset from the web on which we want to train the word vectors. BlazingText expects a single preprocessed text file with space separated tokens and each line of the file should contain a single sentence.In this example, let us train the vectors on [text8](http://mattmahoney.net/dc/textdata.html) dataset (100 MB), which is a small (already preprocessed) version of Wikipedia dump.<jupyter_code>!wget http://mattmahoney.net/dc/text8.zip -O text8.gz
# Uncompressing
!gzip -d text8.gz -f<jupyter_output><empty_output><jupyter_text>After the data downloading and uncompressing is complete, we need to upload it to S3 so that it can be consumed by SageMaker to execute training jobs. We'll use Python SDK to upload these two files to the bucket and prefix location that we have set above.<jupyter_code>train_channel = prefix + '/train'

sess.upload_data(path='text8', bucket=bucket, key_prefix=train_channel)

s3_train_data = 's3://{}/{}'.format(bucket, train_channel)
s3_train_data<jupyter_output><empty_output><jupyter_text>Next we need to setup an output location at S3, where the model artifact will be dumped. These artifacts are also the output of the algorithm's training job.<jupyter_code>s3_output_location = 's3://{}/{}/output'.format(bucket, prefix)
s3_output_location<jupyter_output><empty_output><jupyter_text>Training SetupNow that we are done with all the setup that is needed, we are ready to train our object detector. To begin, let us create a ``sageMaker.estimator.Estimator`` object. This estimator will launch the training job.<jupyter_code>region_name = boto3.Session().region_name
region_name
container = sagemaker.amazon.amazon_estimator.get_image_uri(region_name, "blazingtext", "latest")
print('Using SageMaker BlazingText container: {} ({})'.format(container, region_name))<jupyter_output><empty_output><jupyter_text>Training the BlazingText model for generating word vectors Similar to the original implementation of [Word2Vec](https://arxiv.org/pdf/1301.3781.pdf), SageMaker BlazingText provides an efficient implementation of the continuous bag-of-words (CBOW) and skip-gram architectures using Negative Sampling, on CPUs and additionally on GPU[s]. The GPU implementation uses highly optimized CUDA kernels. To learn more, please refer to [*BlazingText: Scaling and Accelerating Word2Vec using Multiple GPUs*](https://dl.acm.org/citation.cfm?doid=3146347.3146354). BlazingText also supports learning of subword embeddings with CBOW and skip-gram modes. This enables BlazingText to generate vectors for out-of-vocabulary (OOV) words, as demonstrated in this [notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/blazingtext_word2vec_subwords_text8/blazingtext_word2vec_subwords_text8.ipynb). Besides skip-gram and CBOW, SageMaker BlazingText also supports the "Batch Skipgram" mode, which uses efficient mini-batching and matrix-matrix operations ([BLAS Level 3 routines](https://software.intel.com/en-us/mkl-developer-reference-fortran-blas-level-3-routines)). This mode enables distributed word2vec training across multiple CPU nodes, allowing almost linear scale up of word2vec computation to process hundreds of millions of words per second. Please refer to [*Parallelizing Word2Vec in Shared and Distributed Memory*](https://arxiv.org/pdf/1604.04661.pdf) to learn more. BlazingText also supports a *supervised* mode for text classification. It extends the FastText text classifier to leverage GPU acceleration using custom CUDA kernels. The model can be trained on more than a billion words in a couple of minutes using a multi-core CPU or a GPU, while achieving performance on par with the state-of-the-art deep learning text classification algorithms. For more information, please refer to [algorithm documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/blazingtext.html) or [the text classification notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/blazingtext_text_classification_dbpedia/blazingtext_text_classification_dbpedia.ipynb). To summarize, the following modes are supported by BlazingText on different types instances:|          Modes         	| cbow (supports subwords training) 	| skipgram (supports subwords training) 	| batch_skipgram 	| supervised ||:----------------------:	|:----:	|:--------:	|:--------------:	| :--------------:	||   Single CPU instance  	|   ✔  	|     ✔    	|        ✔       	|  ✔  ||   Single GPU instance  	|   ✔  	|     ✔    	|                	|  ✔ (Instance with 1 GPU only)  || Multiple CPU instances 	|      	|          	|        ✔       	|     | |Now, let's define the resource configuration and hyperparameters to train word vectors on *text8* dataset, using "batch_skipgram" mode on two c4.2xlarge instances.<jupyter_code>bt_model = sagemaker.estimator.Estimator(container,
                                         role, 
                                         train_instance_count=2, 
                                         train_instance_type='ml.c4.2xlarge',
                                         train_volume_size = 5,
                                         train_max_run = 360000,
                                         input_mode= 'File',
                                         output_path=s3_output_location,
                                         sagemaker_session=sess)<jupyter_output><empty_output><jupyter_text>Please refer to [algorithm documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/blazingtext_hyperparameters.html) for the complete list of hyperparameters.<jupyter_code>bt_model.set_hyperparameters(mode="skipgram",
                             epochs=5,
                             min_count=5,
                             sampling_threshold=0.0001,
                             learning_rate=0.05,
                             window_size=5,
                             vector_dim=10,
                             negative_samples=5,
                             subwords=True, # Enables learning of subword embeddings for OOV word vector generation
                             min_char=3, # min length of char ngrams
                             max_char=6, # max length of char ngrams
                             batch_size=11, #  = (2*window_size + 1) (Preferred. Used only if mode is batch_skipgram)
                             evaluation=True)# Perform similarity evaluation on WS-353 dataset at the end of training<jupyter_output><empty_output><jupyter_text>Now that the hyper-parameters are setup, let us prepare the handshake between our data channels and the algorithm. To do this, we need to create the `sagemaker.session.s3_input` objects from our data channels. These objects are then put in a simple dictionary, which the algorithm consumes.<jupyter_code>train_data = sagemaker.session.s3_input(s3_train_data, distribution='FullyReplicated', 
                        content_type='text/plain', s3_data_type='S3Prefix')
data_channels = {'train': train_data}<jupyter_output><empty_output><jupyter_text>We have our `Estimator` object, we have set the hyper-parameters for this object and we have our data channels linked with the algorithm. The only  remaining thing to do is to train the algorithm. The following command will train the algorithm. Training the algorithm involves a few steps. Firstly, the instance that we requested while creating the `Estimator` classes is provisioned and is setup with the appropriate libraries. Then, the data from our channels are downloaded into the instance. Once this is done, the training job begins. The provisioning and data downloading will take some time, depending on the size of the data. Therefore it might be a few minutes before we start getting training logs for our training jobs. The data logs will also print out `Spearman's Rho` on some pre-selected validation datasets after the training job has executed. This metric is a proxy for the quality of the algorithm. Once the job has finished a "Job complete" message will be printed. The trained model can be found in the S3 bucket that was setup as `output_path` in the estimator.<jupyter_code>bt_model.fit(inputs=data_channels, logs=True)<jupyter_output><empty_output><jupyter_text>Hosting / InferenceOnce the training is done, we can deploy the trained model as an Amazon SageMaker real-time hosted endpoint. This will allow us to make predictions (or inference) from the model. Note that we don't have to host on the same type of instance that we used to train. Because instance endpoints will be up and running for long, it's advisable to choose a cheaper instance for inference.<jupyter_code>bt_endpoint = bt_model.deploy(initial_instance_count = 1,instance_type = 'ml.m4.xlarge')<jupyter_output><empty_output><jupyter_text>Getting vector representations for words  Use JSON format for inferenceThe payload should contain a list of words with the key as "**instances**". BlazingText supports content-type `application/json`.<jupyter_code>import pandas as pd
import numpy as np
df = pd.read_csv('<csv file name')
df.head()
import nltk

nltk.download('punkt')
nltk.download('stopwords')
nltk.download('averaged_perceptron_tagger')
nltk.download('wordnet')
import nltk, re
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.corpus import wordnet
from nltk.stem import WordNetLemmatizer

# Let's get a list of stop words from the NLTK library
stop = stopwords.words('english')

# These words are important for our problem. We don't want to remove them.
additional_stopwords = ["a", "an", "the", "this", "that", "is", "it", "to", "and"]

stop.extend(additional_stopwords)

# New stop word list
#stop_words = [word for word in stop if word not in excluding]

# Initialize the lemmatizer
wl = WordNetLemmatizer()

# This is a helper function to map NTLK position tags
# Full list is available here: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
def get_wordnet_pos(tag):
    if tag.startswith('J'):
        return wordnet.ADJ
    elif tag.startswith('V'):
        return wordnet.VERB
    elif tag.startswith('N'):
        return wordnet.NOUN
    elif tag.startswith('R'):
        return wordnet.ADV
    else:
        return wordnet.NOUN
    
def process_text_lemmitization(texts): 
    final_text_list=[]
    
    for sent in texts:
        filtered_sentence=[]
        
        if not isinstance(sent,float):
            sent = sent.lower() # Lowercase 
            sent = sent.strip() # Remove leading/trailing whitespace
            sent = re.sub('\s+', ' ', sent) # Remove extra space and tabs
            sent = re.compile('<.*?>').sub('', sent) # Remove HTML tags/markups:

            for w in word_tokenize(sent):
                # We are applying some custom filtering here, feel free to try different things
                # Check if it is not numeric and its length>2 and not in stop words
                if(not w.isnumeric()) and (len(w)>3) and (w not in stop):  
                    # Stem and add to filtered list
                      filtered_sentence.append(w)

            lemmatized_sentence = []

            # Get position tags
            word_pos_tags = nltk.pos_tag(filtered_sentence)

            # Map the position tag and lemmatize the word/token
            for idx, tag in enumerate(word_pos_tags):
                lemmatized_sentence.append(wl.lemmatize(tag[0], get_wordnet_pos(tag[1])))

            lemmatized_text = " ".join(lemmatized_sentence)
            
            final_text_list.append(lemmatized_text)
    
    return final_text_list
df_processed = process_text_lemmitization(df['sentence'])
df_processed
def get_max_word_count(sent_list):
    word_count_list = []
    for sent in sent_list:
        sent_words = word_tokenize(sent)
        word_count = len(sent_words)
        word_count_list.append(word_count)
    return max(word_count_list)
max_word_count = get_max_word_count(df_processed)
max_word_count
max_columns = max_word_count*10
max_columns
def sentence_to_vec2(response):
    sentence_vec = []
    test_array = np.zeros(max_columns)
    for vec in response:
        sentence_vec.extend(vec['vector'])
        sent_array = np.array(sentence_vec)
        test_array[0:sent_array.shape[0]] = sent_array
    return test_array
def process_sent_to_vec(sent_list):
    sent_list_vecs = []
    #print(sent_list)
    for sent in sent_list:
        #print(sent)
        sent_words = word_tokenize(sent)
        payload = {"instances" : sent_words}
        #print(sent_words)
        response = bt_endpoint.predict(json.dumps(payload))
        vecs = json.loads(response)
        sent_vectors = sentence_to_vec2(vecs)
        sent_list_vecs.append(sent_vectors)
    return sent_list_vecs
test_vec = process_sent_to_vec(df_processed)
test_vec_array = np.array(test_vec)
test_vec_array.shape
test_vec_array
train_data = test_vec_array.astype('float32')
np.savetxt("kmeans_train_data.csv", train_data[0:100], delimiter=",")
from sagemaker import KMeans

num_clusters = 5
kmeans = KMeans(role=role,
                train_instance_count=1,
                train_instance_type='ml.c4.xlarge',
                output_path='s3://'+ bucket +'/sentence-similarity/',              
                k=num_clusters)
%%time
kmeans.fit(kmeans.record_set(train_data))
test_channel = prefix + '/batch'

sess.upload_data(path='kmeans_train_data.csv', bucket=bucket, key_prefix=test_channel)
%%time

kmeans_transformer = kmeans.transformer(1, 'ml.m4.xlarge')

# start a transform job
batch_file = 'kmeans_train_data.csv'
input_location = 's3://{}/{}/batch/{}'.format(bucket, prefix, batch_file) # use input data without ID column
kmeans_transformer.transform(input_location, split_type='Line')
kmeans_transformer.wait()
import json
import io
from urllib.parse import urlparse

def get_csv_output_from_s3(s3uri, file_name):
    parsed_url = urlparse(s3uri)
    bucket_name = parsed_url.netloc
    prefix = parsed_url.path[1:]
    s3 = boto3.resource('s3')
    obj = s3.Object(bucket_name, '{}/{}'.format(prefix, file_name))
    return obj.get()["Body"].read().decode('utf-8')
output = get_csv_output_from_s3(kmeans_transformer.output_path, '{}.out'.format(batch_file))
output_df = pd.read_csv(io.StringIO(output), sep=",", header=None)
output_df.head(8)
%%time
kmeans_predictor = kmeans.deploy(initial_instance_count=1, 
                                 instance_type='ml.t2.medium')
%%time
result_kmeans=kmeans_predictor.predict(train_data[0:990])
result_kmeans
cluster_labels = [r.label['closest_cluster'].float32_tensor.values[0] for r in result_kmeans]
cluster_labels
df_results = pd.DataFrame(columns=['student_response'])
df_results['student_response'] = df_processed[0:990]
df_results['cluster'] = cluster_labels
df_results.head()
df_results.to_csv('results_word2vec_sm.csv',index=False)
pd.DataFrame(cluster_labels)[0].value_counts()
import matplotlib.pyplot as plt
import matplotlib
import seaborn as sns
matplotlib.style.use('ggplot')
ax=plt.subplots(figsize=(6,3))
ax=sns.distplot(cluster_labels, kde=False)
title="Histogram of Cluster Counts"
ax.set_title(title, fontsize=12)
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluation Let us now download the word vectors learned by our model and visualize them using a [t-SNE](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding) plot.<jupyter_code>s3 = boto3.resource('s3')

key = bt_model.model_data[bt_model.model_data.find("/", 5)+1:]
s3.Bucket(bucket).download_file(key, 'model.tar.gz')<jupyter_output><empty_output><jupyter_text>Uncompress `model.tar.gz` to get `vectors.txt`<jupyter_code>!tar -xvzf model.tar.gz<jupyter_output><empty_output><jupyter_text>If you set "evaluation" as "true" in the hyperparameters, then "eval.json" will be there in the model artifacts.The quality of trained model is evaluated on word similarity task. We use [WS-353](http://alfonseca.org/eng/research/wordsim353.html), which is one of the most popular test datasets used for this purpose. It contains word pairs together with human-assigned similarity judgments.The word representations are evaluated by ranking the pairs according to their cosine similarities, and measuring the Spearmans rank correlation coefficient with the human judgments.Let's look at the evaluation scores which are there in eval.json. For embeddings trained on the text8 dataset, scores above 0.65 are pretty good.<jupyter_code>!cat eval.json<jupyter_output><empty_output><jupyter_text>Now, let us do a 2D visualization of the word vectors<jupyter_code>import numpy as np
from sklearn.preprocessing import normalize

# Read the 400 most frequent word vectors. The vectors in the file are in descending order of frequency.
num_points = 400

first_line = True
index_to_word = []
with open("vectors.txt","r") as f:
    for line_num, line in enumerate(f):
        if first_line:
            dim = int(line.strip().split()[1])
            word_vecs = np.zeros((num_points, dim), dtype=float)
            first_line = False
            continue
        line = line.strip()
        word = line.split()[0]
        vec = word_vecs[line_num-1]
        for index, vec_val in enumerate(line.split()[1:]):
            vec[index] = float(vec_val)
        index_to_word.append(word)
        if line_num >= num_points:
            break
word_vecs = normalize(word_vecs, copy=False, return_norm=False)
from sklearn.manifold import TSNE

tsne = TSNE(perplexity=40, n_components=2, init='pca', n_iter=10000)
two_d_embeddings = tsne.fit_transform(word_vecs[:num_points])
labels = index_to_word[:num_points]
from matplotlib import pylab
%matplotlib inline

def plot(embeddings, labels):
    pylab.figure(figsize=(20,20))
    for i, label in enumerate(labels):
        x, y = embeddings[i,:]
        pylab.scatter(x, y)
        pylab.annotate(label, xy=(x, y), xytext=(5, 2), textcoords='offset points',
                       ha='right', va='bottom')
    pylab.show()

plot(two_d_embeddings, labels)<jupyter_output><empty_output><jupyter_text>Running the code above might generate a plot like the one below. t-SNE and Word2Vec are stochastic, so although when you run the code the plot won’t look exactly like this, you can still see clusters of similar words such as below where 'british', 'american', 'french', 'english' are near the bottom-left, and 'military', 'army' and 'forces' are all together near the bottom. ![tsne plot of embeddings](./tsne.png)  Stop / Close the Endpoint (Optional)Finally, we should delete the endpoint before we close the notebook.<jupyter_code>sess.delete_endpoint(bt_endpoint.endpoint)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0870202884380347, 'fraction_numerical': 0.010559765338548032, 'mean_word_length': 3.8086506817113306, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 69, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 30, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '496375', 'license': 'Apache-2.0', 'n_tokens_mistral': 6141, 'n_tokens_neox': 5833, 'n_words': 1897, 'path': 'code/SageMaker-word2vec-kmeans.ipynb', 'repo_name': 'awsmrud/Amazon-SageMaker-Text-Similarity-Clustering', 'size': '20455'}