<jupyter_start><jupyter_text>What are we doing with vectors?Vectors have multiple uses in Machine Learning. Apart from getting equivalent numeric representations for ML models, creating word vectors also allows us to measure similarity of given documents. These vectors also uses the same techniques used in Vector Algebra like Euclidean Distance, Cosine Similarity, etc.**Why Cosine Similarity?**A commonly used approach to match similar documents is based on counting the maximum number of common words between the documents. But, as the size of the document increases, the number of common words tend to increase even if the documents talk about different topics.Cosine similarity is a metric used to determine how similar the documents are irrespective of their size.If there are two vectors at in a multi-dimensional space, then the angle between them is given by the cosine similarity. The distance might be large (Euclidean Distance) but the angle could be smaller which indicates higher similarity.Mathematical Formula -  ```u.v / |u||v| , where u, v are two vectors```Read more - https://www.machinelearningplus.com/nlp/cosine-similarity/<jupyter_code>doc_trump = "Mr. Trump became president after winning the political election. Though he lost the support of some republican friends, Trump is friends with President Putin"

doc_election = "President Trump says Putin had no political interference is the election outcome. He says it was a witchhunt by political parties. He claimed President Putin is a friend who had nothing to do with the election"

doc_putin = "Post elections, <NAME> became President of Russia. President Putin had served as the Prime Minister earlier in his political career"

documents = [doc_trump, doc_election, doc_putin]
from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

count_vectorizer = CountVectorizer(stop_words='english')
sparse_matrix = count_vectorizer.fit_transform(documents) # converts to a matrix of word vectors

doc_term_matrix = sparse_matrix.toarray()

df = pd.DataFrame(doc_term_matrix, 
                  columns=count_vectorizer.get_feature_names(), 
                  index=['doc_trump', 'doc_election', 'doc_putin'])
df
from sklearn.metrics.pairwise import cosine_similarity

print(cosine_similarity(df, df))<jupyter_output>[[1.         0.51639778 0.36893239]
 [0.51639778 1.         0.45360921]
 [0.36893239 0.45360921 1.        ]]<jupyter_text>Document SimilarityUsing Cosine Similarity we can understand how similar are two documents.<jupyter_code>import nltk
import numpy as np
# nltk.download('punkt')
from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(stop_words='english')

def cosine_sim(text1, text2):
    tfidf = vectorizer.fit_transform([text1, text2])
    print('Vocab: {}'.format(vectorizer.vocabulary_))
    vec1 = tfidf.toarray()[0]
    vec2 = tfidf.toarray()[1]
    dot_prod = np.dot(vec1, vec2)
    normed = np.linalg.norm(vec1)* np.linalg.norm(vec2)
    return dot_prod/normed
cosine_sim('a little bird', 'a little bird')
cosine_sim('a little bird', 'a little bird chirps')
cosine_sim('a little bird', 'a big dog barks')<jupyter_output>Vocab: {'little': 4, 'bird': 2, 'big': 1, 'dog': 3, 'barks': 0}
{'fraction_non_alphanumeric': 0.07858243451463791, 'fraction_numerical': 0.02280431432973806, 'mean_word_length': 5.0, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '471909', 'license': 'MIT', 'n_tokens_mistral': 990, 'n_tokens_neox': 917, 'n_words': 362, 'path': 'Week 4/2_Document Similarity.ipynb', 'repo_name': 'royn5618/NLP_Workshop_WAIA_Sept_2021', 'size': '3253'}