<jupyter_start><jupyter_text>The similarity methodThe Doc and Span objects also have vectors. The vector of a sentence or a span is the average of its words' vectors<jupyter_code>doc1 = nlp("I visited England.")
doc2 = nlp("I went to London.")
doc1[1:3].similarity(doc2[1:4])
doc1[2].similarity(doc2[3])
doc1.similarity(doc2)
doc1.similarity(doc1)
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
vocab = nlp("cat dog tiger elephant bird monkey lion cheetah burger pizza food cheese wine salad noodles macaroni fruit vegetable")
words = [word.text for word in vocab]
vectors = np.vstack([word.vector for word in vocab if word.has_vector])
pca = PCA(n_components=2)
vectors_transformed = pca.fit_transform(vectors)
plt.figure(figsize=(16,12))
plt.scatter(vectors_transformed[:,0], vectors_transformed[:,1])
for word, coord in zip(words, vectors_transformed):
    x,y = coord
    plt.text(x,y,word, size=15)
plt.show()<jupyter_output><empty_output><jupyter_text>Using third-party word vectors<jupyter_code>!wget https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M-subword.vec.zip
!unzip wiki-news-300d-1M-subword.vec.zip
!python -m spacy init vectors en wiki-news-300d-1M-subword.vec en_subwords_wiki_lg
nlp = spacy.load("en_subwords_wiki_lg")
doc = nlp("I want Mango.")
doc[1].vector
nlp.pipe_names<jupyter_output><empty_output><jupyter_text>This model is blank. it does not contain any other pipeline components Semantic SimilarityspaCy uses cosine similarity to calculate semantic similarity. Hence, calling the similarity method helps us make cosine similarity calculations.<jupyter_code>nlp = spacy.load("en_core_web_md")
sentences = nlp("I purchased a science fiction book last week. I loved everything related to this fragrance: light, floral and feminine... I purchased a bottle of wine.")
key = nlp("perfume")
for sent in sentences.sents:
    print(sent.similarity(key))<jupyter_output>0.2481654331382154
0.5075297559861377
0.4215429463030136<jupyter_text>Extracting important words<jupyter_code>for sent in sentences.sents:
  nchunks = [nchunk.text for nchunk in sent.noun_chunks]
  nchunk_doc = nlp(" ".join(nchunks))
  print(nchunk_doc.similarity(key))<jupyter_output>0.21390893517254456
0.6370830556977022
0.44506391511570403<jupyter_text>Extracting and comparing named entities<jupyter_code>doc1 = nlp("Google Search, often referred as Google, is the most popular search engine nowadays. It answers a huge volume of queries every day.")
doc2 = nlp("Microsoft Bing is another popular search engine. Microsoft is known by its star product Microsoft Windows, a popular operating system sold over the world.")
doc3 = nlp("The Dead Sea is the lowest lake in the world, located in the Jordan Valley of Israel. It is also the saltiest lake in the world.")
print(doc1.ents)
print(doc2.ents)
print(doc3.ents)
ents1 = [ent.text for ent in doc1.ents]
ents2 = [ent.text for ent in doc2.ents]
ents3 = [ent.text for ent in doc3.ents]
ents1 = nlp(" ".join(ents1))
ents2 = nlp(" ".join(ents2))
ents3 = nlp(" ".join(ents3))
print(ents1.similarity(ents2))
print(ents1.similarity(ents3))
print(ents2.similarity(ents3))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10365282547611614, 'fraction_numerical': 0.05182641273805807, 'mean_word_length': 6.1042128603104215, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '184838', 'license': 'MIT', 'n_tokens_mistral': 1183, 'n_tokens_neox': 1076, 'n_words': 317, 'path': 'Exploring Spacy3/Word_Vectors_and_Semantic_Similarity.ipynb', 'repo_name': 'luckyrabbit85/NLP', 'size': '3203'}