<jupyter_start><jupyter_text>Word2Vec IntroUnderstand how word2vec algorithm works[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/elephantscale/cool-ML-demos/blob/main/nlp/word2vec-1-intro.ipynb) Runtime15 minsreferences- https://machinelearningmastery.com/develop-word-embeddings-python-gensim/ OverviewCheckout [projector.tensorflow.org](https://projector.tensorflow.org/)  for online demo!  Step 1 - Data<jupyter_code>## Here we are skipping stop words like : is, are
sentences = [
        'dog  cat  domestic animals'.split(),   # 'dog and cat are domestic animals'
        'dog  cat  pets'.split(),   # dog and cat are pets
        'dog  cat  enemies'.split(),  # dog and cat are enemies
        'sushi fish'.split(),  # sushi has fish
        'sushi  rice'.split()  # sushi has rice
]<jupyter_output><empty_output><jupyter_text>Step 2 - Train a Word2Vec model<jupyter_code>%%time
from gensim.models import Word2Vec

# train
model = Word2Vec(sentences, min_count=1)
# summarize the loaded model
print(model)
# summarize vocabulary
model.wv.vocab
## access vector for one word
print(model.wv['dog'])<jupyter_output>[-4.8351591e-03  3.4744530e-03 -1.4519688e-03 -3.2536306e-03
 -3.7812891e-03 -2.9908826e-03  2.8983282e-03 -8.0656190e-04
  2.8796170e-03 -1.7652292e-03 -2.5155430e-03  3.0750860e-03
  1.1371209e-03  1.3201663e-03 -3.0576226e-03  2.0660197e-03
  1.2492742e-03  1.4134261e-03 -1.8425760e-04  2.0438877e-03
 -1.9448301e-03  8.3302111e-05 -1.5790496e-03  3.8306194e-03
 -4.4419779e-03 -2.3121426e-03 -1.6115182e-03 -3.7887241e-03
  6.5071322e-04 -1.0384436e-03 -3.8295430e-03  2.4209949e-03
 -4.8135202e-03  2.4192906e-03  3.5803777e-03  5.2367459e-04
  1.4203154e-03  2.8078414e-03 -3.0081666e-03 -1.7360783e-03
  3.8228966e-03 -1.0841951e-03 -1.4380576e-03  5.7306909e-04
 -2.7666714e-03  3.8534575e-04 -6.5972086e-04  1.3162927e-03
  1.5718110e-03 -1.4500398e-03 -1.1755885e-03 -1.1756103e-03
  3.1252897e-03 -3.7989309e-03  3.7095458e-03 -2.6361065e-03
 -2.1640293e-03  3.0332084e-03 -5.8502116e-04  2.4220461e-03
 -4.8750886e-03 -2.4334404e-03 -8.3359348e-04 -4.2625610e-03
 -3.9423630e-03 -1.51917[...]<jupyter_text>Step 3 - Plot the modelHere we are reducing the model from 100 dimensions to just 2, so we can plot.  We are using PCA algorithm to reduce the dimensions.Does this plot makes sense? :-)  We don't have much data !<jupyter_code>%matplotlib inline
from matplotlib import pyplot
from gensim.models import Word2Vec
from sklearn.decomposition import PCA


# fit a 2d PCA model to the vectors
X = model[model.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model.wv.vocab)
for i, word in enumerate(words):
  pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()<jupyter_output>/home/ubuntu/apps/anaconda/lib/python3.7/site-packages/ipykernel_launcher.py:8: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).<jupyter_text>Step 4 - Find Similarities<jupyter_code>model.wv.most_similar('cat')
## does the result make sense? :-) 
model.wv.most_similar('rice')<jupyter_output><empty_output><jupyter_text>Step 5 - More 'complicated' QueriesHere we can ask more complex queries.  The results are fun<jupyter_code>model.wv.most_similar(positive=['rice', 'sushi'])
 model.wv.most_similar(positive=['cat', 'dog'], negative=['fish'], topn=1)<jupyter_output><empty_output><jupyter_text>Step 6 - Save the model<jupyter_code># save
model.save('my_model.bin')
print ("saved")
## load the saved model
new_model = Word2Vec.load('my_model.bin')
print(new_model)
new_model.wv.vocab
## query the new model
new_model.wv.most_similar('fish')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13781468985206333, 'fraction_numerical': 0.1793407734233065, 'mean_word_length': 4.293956043956044, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '619751', 'license': 'MIT', 'n_tokens_mistral': 1962, 'n_tokens_neox': 1587, 'n_words': 310, 'path': 'nlp/word2vec-1-intro.ipynb', 'repo_name': 'elephantscale/cool-ML-demos', 'size': '3853'}