<jupyter_start><jupyter_text>Developing Word Embeddings Rather than use pre-trained embeddings (as we did in the sentence similarity baseline_deep_dive [notebook](../sentence_similarity/baseline_deep_dive.ipynb)), we can train word embeddings using our own dataset. In this notebook, we demonstrate the training process for producing word embeddings using the word2vec, GloVe, and fastText models. We'll utilize the STS Benchmark dataset for this task.   Table of Contents* [Data Loading and Preprocessing](Load-and-Preprocess-Data)* [Word2Vec](Word2Vec)* [fastText](fastText)* [GloVe](GloVe)* [Concluding Remarks](Concluding-Remarks)<jupyter_code>import gensim
import sys
import os

# Set the environment path
sys.path.append("../..")

import numpy as np
from utils_nlp.dataset.preprocess import (
    to_lowercase,
    to_spacy_tokens,
    rm_spacy_stopwords,
)
from utils_nlp.dataset import stsbenchmark
from utils_nlp.common.timer import Timer
from gensim.models import Word2Vec
from gensim.models.fasttext import FastText
# Set the path for where your repo is located
NLP_REPO_PATH = os.path.join('..','..')

# Set the path for where your datasets are located
BASE_DATA_PATH = os.path.join(NLP_REPO_PATH, "data")

# Set the path for location to save embeddings
SAVE_FILES_PATH = os.path.join(BASE_DATA_PATH, "trained_word_embeddings")
if not os.path.exists(SAVE_FILES_PATH):
    os.makedirs(SAVE_FILES_PATH)<jupyter_output><empty_output><jupyter_text>Load and Preprocess Data<jupyter_code># Produce a pandas dataframe for the training set
train_raw = stsbenchmark.load_pandas_df(BASE_DATA_PATH, file_split="train")

# Clean the sts dataset
sts_train = stsbenchmark.clean_sts(train_raw)
sts_train.head(5)
# Check the size of our dataframe
sts_train.shape<jupyter_output><empty_output><jupyter_text>Training set preprocessing<jupyter_code># Convert all text to lowercase
df_low = to_lowercase(sts_train)  
# Tokenize text
sts_tokenize = to_spacy_tokens(df_low) 
# Tokenize with removal of stopwords
sts_train_stop = rm_spacy_stopwords(sts_tokenize) 
# Append together the two sentence columns to get a list of all tokenized sentences.
all_sentences =  sts_train_stop[["sentence1_tokens_rm_stopwords", "sentence2_tokens_rm_stopwords"]]
# Flatten two columns into one list and remove all sentences that are size 0 after tokenization and stop word removal.
sentences = [i for i in all_sentences.values.flatten().tolist() if len(i) > 0]
len(sentences)
sentence_lengths = [len(i) for i in sentences]
print("Minimum sentence length is {} tokens".format(min(sentence_lengths)))
print("Maximum sentence length is {} tokens".format(max(sentence_lengths)))
print("Median sentence length is {} tokens".format(np.median(sentence_lengths)))
sentences[:10]<jupyter_output><empty_output><jupyter_text>Word2Vec Word2vec is a predictive model for learning word embeddings from text (see [original research paper](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)). Word embeddings are learned such that words that share common contexts in the corpus will be close together in the vector space. There are two different model architectures that can be used to produce word2vec embeddings: continuous bag-of-words (CBOW) or continuous skip-gram. The former uses a window of surrounding words (the "context") to predict the current word and the latter uses the current word to predict the surrounding context words. See this [tutorial](https://www.guru99.com/word-embedding-word2vec.html3) on word2vec for more detailed background on the model. The gensim Word2Vec model has many different parameters (see [here](https://radimrehurek.com/gensim/models/word2vec.htmlgensim.models.word2vec.Word2Vec)) but the ones that are useful to know about are:  - size: length of the word embedding/vector (defaults to 100)- window: maximum distance between the word being predicted and the current word (defaults to 5)- min_count: ignores all words that have a frequency lower than this value (defaults to 5)- workers: number of worker threads used to train the model (defaults to 3)- sg: training algorithm; 1 for skip-gram and 0 for CBOW (defaults to 0)<jupyter_code># Set up a Timer to see how long the model takes to train
t = Timer()
t.start()

# Train the Word2vec model
word2vec_model = Word2Vec(sentences, size=100, window=5, min_count=5, workers=3, sg=0)

t.stop()
print("Time elapsed: {}".format(t))<jupyter_output>Time elapsed: 0.3194<jupyter_text>Now that the model is trained we can:1. Query for the word embeddings of a given word. 2. Inspect the model vocabulary3. Save the word embeddings<jupyter_code># 1. Let's see the word embedding for "apple" by accessing the "wv" attribute and passing in "apple" as the key.
print("Embedding for apple:", word2vec_model.wv["apple"])

# 2. Inspect the model vocabulary by accessing keys of the "wv.vocab" attribute. We'll print the first 20 words.
print("\nFirst 30 vocabulary words:", list(word2vec_model.wv.vocab)[:20])

# 3. Save the word embeddings. We can save as binary format (to save space) or ASCII format.
word2vec_model.wv.save_word2vec_format(SAVE_FILES_PATH+"word2vec_model", binary=True)  # binary format
word2vec_model.wv.save_word2vec_format(SAVE_FILES_PATH+"word2vec_model", binary=False)  # ASCII format<jupyter_output>Embedding for apple: [-0.13886626 -0.04330257  0.12527628  0.08564945  0.02040523 -0.10037457
 -0.1182736   0.05916803 -0.09810918  0.11094606 -0.00045659 -0.07130833
 -0.07526248  0.01439941 -0.01924936 -0.04267681  0.05364342  0.01334886
  0.09927388  0.04298429  0.07616432 -0.09218667  0.13563654  0.13954957
  0.17032589  0.13070972  0.04971378  0.05326121  0.1633883   0.0867981
  0.01025774  0.19571003 -0.11564688  0.00285543 -0.02306972 -0.07086422
 -0.03311775  0.16642122  0.10450041  0.11148815 -0.11674852 -0.10021858
 -0.00149789 -0.10769422  0.1467818  -0.00330875  0.09308671 -0.12129212
  0.07261119  0.07583102  0.00192156  0.23766024 -0.0063716  -0.10565527
 -0.06545153  0.04053855  0.24339062  0.15191206 -0.04718588 -0.05213067
  0.00187512 -0.08648538 -0.05337012  0.15507293 -0.09485061  0.03063929
  0.00369516 -0.20911641  0.09312427  0.03583751  0.07270095  0.18968543
  0.08637197 -0.03679648  0.12222783 -0.11879333 -0.1462169   0.02210324
  0.18023533  0.03193852 -0.025[...]<jupyter_text>fastText fastText is an unsupervised algorithm created by Facebook Research for efficiently learning word embeddings (see [original research paper](https://arxiv.org/pdf/1607.04606.pdf)). fastText is significantly different than word2vec or GloVe in that these two algorithms treat each word as the smallest possible unit to find an embedding for. Conversely, fastText assumes that words are formed by an n-gram of characters (i.e. 2-grams of the word "language" would be {la, an, ng, gu, ua, ag, ge}). The embedding for a word is then composed of the sum of these character n-grams. This has advantages when finding word embeddings for rare words and words not present in the dictionary, as these words can still be broken down into character n-grams. Typically, for smaller datasets, fastText performs better than word2vec or GloVe. See this [tutorial](https://fasttext.cc/docs/en/unsupervised-tutorial.html) on fastText for more detail. The gensim fastText model has many different parameters (see [here](https://radimrehurek.com/gensim/models/fasttext.htmlgensim.models.fasttext.FastText)) but the ones that are useful to know about are:  - size: length of the word embedding/vector (defaults to 100)- window: maximum distance between the word being predicted and the current word (defaults to 5)- min_count: ignores all words that have a frequency lower than this value (defaults to 5)- workers: number of worker threads used to train the model (defaults to 3)- sg: training algorithm- 1 for skip-gram and 0 for CBOW (defaults to 0)- iter: number of epochs (defaults to 5)<jupyter_code># Set up a Timer to see how long the model takes to train
t = Timer()
t.start()

# Train the FastText model
fastText_model = FastText(size=100, window=5, min_count=5, sentences=sentences, iter=5)

t.stop()
print("Time elapsed: {}".format(t))<jupyter_output>Time elapsed: 9.3665<jupyter_text>We can utilize the same attributes as we saw above for word2vec due to them both originating from the gensim package<jupyter_code># 1. Let's see the word embedding for "apple" by accessing the "wv" attribute and passing in "apple" as the key.
print("Embedding for apple:", fastText_model.wv["apple"])

# 2. Inspect the model vocabulary by accessing keys of the "wv.vocab" attribute. We'll print the first 20 words.
print("\nFirst 30 vocabulary words:", list(fastText_model.wv.vocab)[:20])

# 3. Save the word embeddings. We can save as binary format (to save space) or ASCII format.
fastText_model.wv.save_word2vec_format(SAVE_FILES_PATH+"fastText_model", binary=True)  # binary format
fastText_model.wv.save_word2vec_format(SAVE_FILES_PATH+"fastText_model", binary=False)  # ASCII format<jupyter_output>Embedding for apple: [-0.2255679  -0.15831569  0.03804937  0.47731966  0.47977886 -0.27653983
 -0.27343377 -0.4507852  -0.05649747  0.01470412  0.27904618 -0.02155268
 -0.02492249 -0.07855172  0.18532543  0.25709668  0.05939932  0.10333744
 -0.09892524 -0.61932683 -0.15273307 -0.02246136 -0.06295346 -0.5022594
 -0.13407618 -0.10411069  0.13370538  0.11902415 -0.44436237  0.27073038
  0.06540621 -0.02650584 -0.0179158   0.08797703  0.18899101  0.12898529
  0.05865225 -0.18658654 -0.40497953 -0.23991017  0.30457255  0.39893195
  0.2913193  -0.18734889  0.10662938 -0.1165131  -0.42884877  0.31400812
  0.04840293  0.10146416 -0.10285722 -0.21854313 -0.69022155 -0.48051542
 -0.17416449  0.12879132  0.12302257 -0.32911557 -0.48828328  0.22531843
 -0.35535514 -0.34300882  0.07264371  0.262703   -0.10182904  0.03486007
 -0.09019874  0.12621203  0.35632437 -0.10350075  0.3397234  -0.04080832
 -0.17116521 -0.20685913  0.18177888  0.19674565  0.00776504 -0.22853185
  0.01387324 -0.33452377  0.101[...]<jupyter_text>GloVe GloVe is an unsupervised algorithm for obtaining word embeddings created by the Stanford NLP group (see [original research paper](https://nlp.stanford.edu/pubs/glove.pdf)). Training occurs on word-word co-occurrence statistics with the objective of learning word embeddings such that the dot product of two words' embeddings is equal to the words' probability of co-occurrence. See this [tutorial](https://nlp.stanford.edu/projects/glove/) on GloVe for more detailed background on the model. Gensim doesn't have an implementation of the GloVe model and the other python packages that implement GloVe are unstable, so we leveraged the code directly from the Stanford NLP [repo](https://github.com/stanfordnlp/GloVe).<jupyter_code># Define path
glove_model_path = os.path.join(NLP_REPO_PATH, "utils_nlp", "models", "glove")
# Execute shell commands
!cd $glove_model_path && make<jupyter_output>mkdir -p build
gcc src/glove.c -o build/glove -lm -pthread -Ofast -march=native -funroll-loops -Wall -Wextra -Wpedantic
[01m[Ksrc/glove.c:[m[K In function ‘[01m[Kglove_thread[m[K’:
[01m[Ksrc/glove.c:117:9:[m[K [01;35m[Kwarning: [m[Kignoring return value of ‘[01m[Kfread[m[K’, declared with attribute warn_unused_result [-Wunused-result]
         fread(&cr, sizeof(CREC), 1, fin);
[01;32m[K         ^[m[K
gcc src/shuffle.c -o build/shuffle -lm -pthread -Ofast -march=native -funroll-loops -Wall -Wextra -Wpedantic
[01m[Ksrc/shuffle.c:[m[K In function ‘[01m[Kshuffle_merge[m[K’:
[01m[Ksrc/shuffle.c:106:17:[m[K [01;35m[Kwarning: [m[Kignoring return value of ‘[01m[Kfread[m[K’, declared with attribute warn_unused_result [-Wunused-result]
                 fread(&array[i], sizeof(CREC), 1, fid[j]);
[01;32m[K                 ^[m[K
[01m[Ksrc/shuffle.c:[m[K In function ‘[01m[Kshuffle_by_chunks[m[K’:
[01m[Ksrc/shuffle.c:163:9:[m[K [01;35m[Kwa[...]<jupyter_text>Train GloVe vectors Training GloVe embeddings requires some data prep and then 4 steps (also documented in the original Stanford NLP repo [here](https://github.com/stanfordnlp/GloVe/tree/master/src)). **Step 0: Prepare Data**   In order to train our GloVe vectors, we first need to save our corpus as a text file with all words separated by 1+ spaces or tabs. Each document/sentence is separated by a new line character.<jupyter_code># Save our corpus as tokens delimited by spaces with new line characters in between sentences.
training_corpus_file_path = os.path.join(SAVE_FILES_PATH, "training-corpus-cleaned.txt")
with open(training_corpus_file_path, 'w', encoding='utf8') as file:
    for sent in sentences:
        file.write(" ".join(sent) + "\n")
# Set up a Timer to see how long the model takes to train
t = Timer()
t.start()<jupyter_output><empty_output><jupyter_text>**Step 1: Build Vocabulary**Run the vocab_count executable. There are 3 optional parameters:1. min-count: lower limit on how many times a word must appear in dataset. Otherwise the word is discarded from our vocabulary.2. max-vocab: upper bound on the number of vocabulary words to keep3. verbose: 0, 1, or 2 (default)Then provide the path to the text file we created in Step 0 followed by a file path that we'll save the vocabulary to<jupyter_code># Define path
vocab_count_exe_path = os.path.join(glove_model_path, "build", "vocab_count")
vocab_file_path = os.path.join(SAVE_FILES_PATH, "vocab.txt")
# Execute shell commands
!$vocab_count_exe_path -min-count 5 -verbose 2 <$training_corpus_file_path> $vocab_file_path<jupyter_output>BUILDING VOCABULARY
Processed 0 tokens.[0GProcessed 85334 tokens.
Counted 11716 unique words.
Truncating vocabulary at min count 5.
Using vocabulary of size 2943.<jupyter_text>**Step 2: Construct Word Co-occurrence Statistics**Run the cooccur executable. There are many optional parameters, but we list the top ones here:1. symmetric: 0 for only looking at left context, 1 (default) for looking at both left and right context2. window-size: number of context words to use (default 15)3. verbose: 0, 1, or 2 (default)4. vocab-file: path/name of the vocabulary file created in Step 15. memory: soft limit for memory consumption, default 46. max-product: limit the size of dense co-occurrence array by specifying the max product (integer) of the frequency counts of the two co-occurring wordsThen provide the path to the text file we created in Step 0 followed by a file path that we'll save the co-occurrences to<jupyter_code># Define path
cooccur_exe_path = os.path.join(glove_model_path, "build", "cooccur")
cooccurrence_file_path = os.path.join(SAVE_FILES_PATH, "cooccurrence.bin")
# Execute shell commands
!$cooccur_exe_path -memory 4 -vocab-file $vocab_file_path -verbose 2 -window-size 15 <$training_corpus_file_path> $cooccurrence_file_path<jupyter_output>COUNTING COOCCURRENCES
window size: 15
context: symmetric
max product: 13752509
overflow length: 38028356
Reading vocab from file "../../data/trained_word_embeddings/vocab.txt"...loaded 2943 words.
Building lookup table...table contains 8661250 elements.
Processing token: 0[0GProcessed 85334 tokens.
Writing cooccurrences to disk......2 files in total.
Merging cooccurrence files: processed 0 lines.[39G0 lines.[39G100000 lines.[0GMerging cooccurrence files: processed 188154 lines.<jupyter_text>**Step 3: Shuffle the Co-occurrences**Run the shuffle executable. The parameters are as follows:1. verbose: 0, 1, or 2 (default)2. memory: soft limit for memory consumption, default 43. array-size: limit to the length of the buffer which stores chunks of data to shuffle before writing to diskThen provide the path to the co-occurrence file we created in Step 2 followed by a file path that we'll save the shuffled co-occurrences to<jupyter_code># Define path
shuffle_exe_path = os.path.join(glove_model_path, "build", "shuffle")
cooccurrence_shuf_file_path = os.path.join(SAVE_FILES_PATH, "cooccurrence.shuf.bin")
# Execute shell commands
!$shuffle_exe_path -memory 4 -verbose 2 <$cooccurrence_file_path> $cooccurrence_shuf_file_path<jupyter_output>SHUFFLING COOCCURRENCES
array size: 255013683
Shuffling by chunks: processed 0 lines.[22Gprocessed 188154 lines.
Wrote 1 temporary file(s).
Merging temp files: processed 0 lines.[31G188154 lines.[0GMerging temp files: processed 188154 lines.<jupyter_text>**Step 4: Train GloVe model**Run the glove executable. There are many parameter options, but the top ones are listed below:1. verbose: 0, 1, or 2 (default)2. vector-size: dimension of word embeddings (50 is default)3. threads: number threads, default 84. iter: number of iterations, default 255. eta: learning rate, default 0.056. binary: whether to save binary format (0: text = default, 1: binary, 2: both)7. x-max: cutoff for weighting function, default is 1008. vocab-file: file containing vocabulary as produced in Step 19. save-file: filename to save vectors to 10. input-file: filename with co-occurrences as returned from Step 3<jupyter_code># Define path
glove_exe_path = os.path.join(glove_model_path, "build", "glove")
glove_vector_file_path = os.path.join(SAVE_FILES_PATH, "GloVe_vectors")
# Execute shell commands
!$glove_exe_path -save-file $glove_vector_file_path -threads 8 -input-file \
$cooccurrence_shuf_file_path -x-max 10 -iter 15 -vector-size 50 -binary 2 \
-vocab-file $vocab_file_path -verbose 2
t.stop()
print("Time elapsed: {}".format(t))<jupyter_output>Time elapsed: 3.4293<jupyter_text>Inspect Word Vectors Like we did above for the word2vec and fastText models, let's now inspect our word embeddings<jupyter_code>#load in the saved word vectors.
glove_wv = {}
glove_vector_txt_file_path = os.path.join(SAVE_FILES_PATH, "GloVe_vectors.txt")
with open(glove_vector_txt_file_path, encoding='utf-8') as f:
    for line in f:
        split_line = line.split(" ")
        glove_wv[split_line[0]] = [float(i) for i in split_line[1:]]
# 1. Let's see the word embedding for "apple" by passing in "apple" as the key.
print("Embedding for apple:", glove_wv["apple"])

# 2. Inspect the model vocabulary by accessing keys of the "wv.vocab" attribute. We'll print the first 20 words.
print("\nFirst 30 vocabulary words:", list(glove_wv.keys())[:20])<jupyter_output>Embedding for apple: [-0.037004, -0.000665, -0.028638, 0.025758, -0.050187, 0.038694, 0.016966, -0.042032, -0.033963, 0.143667, -0.068749, -0.005046, 0.180022, 0.088593, -0.04615, -0.013351, 0.064172, 0.051637, -0.000885, 0.009899, -0.092548, -0.026595, 0.036515, -0.09158, -0.027992, 0.016924, -0.024003, -0.029879, 0.252747, 0.093754, -0.034897, 0.079439, -0.073516, -0.110923, 0.095652, 0.072123, -0.047069, -0.17929, -0.068377, -0.224694, -0.016158, 0.236704, 0.010695, -0.133073, 0.084929, 0.102969, 0.040056, -0.009444, -0.051333, 0.130339]

First 30 vocabulary words: ['.', ',', 'man', '-', '"', 'woman', "'", 'said', 'dog', 'playing', ':', 'white', 'black', '$', 'killed', 'percent', 'new', 'syria', 'people', 'china']
{'fraction_non_alphanumeric': 0.10493013445821249, 'fraction_numerical': 0.11426311626680728, 'mean_word_length': 4.949184441656211, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 51, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '242869', 'license': 'MIT', 'n_tokens_mistral': 7910, 'n_tokens_neox': 6426, 'n_words': 2074, 'path': 'examples/embeddings/embedding_trainer.ipynb', 'repo_name': 'Irina-64/nlp-recipes', 'size': '18965'}