<jupyter_start><jupyter_text>Install Spacy:<jupyter_code>!python -m spacy download en_core_web_lg
# Small model do not have word2vec inbuilt, it is present in spacy large<jupyter_output>Collecting en_core_web_lg==2.2.5
[?25l  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.2.5/en_core_web_lg-2.2.5.tar.gz (827.9MB)
[K     |████████████████████████████████| 827.9MB 1.3MB/s 
[?25hRequirement already satisfied: spacy>=2.2.2 in /usr/local/lib/python3.7/dist-packages (from en_core_web_lg==2.2.5) (2.2.4)
Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (0.8.2)
Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (7.4.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (57.0.0)
Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (1.1.3)
Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/[...]<jupyter_text>Additional Reading Reference for Word2Vec:* http://jalammar.github.io/illustrated-word2vec  Word2Vec with Spacy:<jupyter_code>!python -m spacy download en
# en model has 1 million unique vectors
import spacy
print(spacy.__version__) 
import en_core_web_lg
nlp = en_core_web_lg.load()
# en_core_web_lg.load() works fine, but spacy.load("en_core_web_lg") doesn't
word = "april sunflower dog squirrel dljkbkdsjh cat lion"
doc = nlp(word)

for token in doc:
  print(token.text, token.has_vector, token.vector_norm)<jupyter_output>april True 6.337781
sunflower True 6.952784
dog True 7.0336733
squirrel True 6.4247537
dljkbkdsjh False 0.0
cat True 6.6808186
lion True 6.5120897<jupyter_text>Semantic Similarity: Checking similarity of 2 objects, predicting similarity is useful for building recommendation systems or duplicate flagging.<jupyter_code>for token in doc:
  for token_2 in doc:
    print(token.text, " || ", token_2.text, " || ",token.similarity(token_2)) 
nlp(u'lion').vector
nlp(u'He played golf everyday').vector
nlp(u'He played golf everyday').vector.shape
# 300 dimensions for the vector for this document
nlp(u'lion').vector.shape
nlp.vocab.vectors.shape
tokens = nlp("Dog Cat Souparna")

for token in tokens:
  print(token.text, " || ", token.has_vector, " || ",token.is_oov, " || ",token.vector_norm)<jupyter_output>Dog  ||  True  ||  False  ||  7.0336733
Cat  ||  True  ||  False  ||  6.6808186
Souparna  ||  False  ||  True  ||  0.0<jupyter_text>Cosine Similarity:<jupyter_code>from scipy import spatial

cosine_sim = lambda vec1, vec2: 1 - spatial.distance.cosine(vec1, vec2)
king = nlp.vocab["king"].vector
man = nlp.vocab["man"].vector
woman = nlp.vocab["woman"].vector
# king - man + woman ==> queen/princess...
new_vec = king - man + woman
computed_sim = []

for word in nlp.vocab:
  if word.has_vector:
    if word.is_lower:
      if word.is_alpha:
        similarity = cosine_sim(new_vec, word.vector)
        computed_sim.append((word, similarity))
computed_sim = sorted(computed_sim, key=lambda item: -item[1])
[t[0].text for t in computed_sim[:10]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15045317220543808, 'fraction_numerical': 0.05709969788519637, 'mean_word_length': 4.880994671403197, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 27, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '160448', 'license': 'MIT', 'n_tokens_mistral': 1395, 'n_tokens_neox': 1297, 'n_words': 267, 'path': 'Word2Vec_Spacy.ipynb', 'repo_name': 'souparnabose99/Word2Vec-Spacy', 'size': '3310'}