<jupyter_start><jupyter_text>WMD And Cosine Similarity General.Ejemplos GENERALES de WDM y Cosine Similarity:1. Ejemplo 1 WMD basado en https://towardsdatascience.com/word-distance-between-word-embeddings-cc3e9cf1d632 con código fuente en https://github.com/makcedward/nlp/blob/master/sample/nlp-word_mover_distance.ipynb2. Ejemplo 2 Cosine Similarity & TF IDF de https://leantechblog.wordpress.com/2020/08/23/como-estimar-la-similitud-entre-documentos-con-python/ con código fuente en https://github.com/cjcarvajal/text-similarity-obama-trump3. Ejemplo 3 WMD using Spacy basado en https://stackoverflow.com/questions/54535535/how-to-improve-word-mover-distance-similarity-in-python-and-provide-similarity-s (el código es el de la 1ra respuesta).4. Ejemplo 4 WMD BIEN completo: https://github.com/Seif-Tarek/Document-Similarity-using-Word-Mover-Distance/blob/master/WMD_TextSimilarity.ipynb5. Keyword extraction using TF*IDF: https://www.analyticsvidhya.com/blog/2020/11/words-that-matter-a-simple-guide-to-keyword-extraction-in-python/  Ejemplo 1 WMD  Word Mover's Distance (WMD) is proposed fro distance measurement between 2 documents (or sentences). It leverages Word Embeddings power to overcome those basic distance measurement limitations. WMD was introduced by Kusner et al. in 2015. Instead of using Euclidean Distance and other bag-of-words based distance measurement, they proposed to use word embeddings to calculate the similarities. To be precise, it uses normalized Bag-of-Words and Word Embeddings to calculate the distance between documents.In the previous blog, I shared how we can use simple way to find the "similarity" between two documents (or sentences). At that time, Euclidean Distance, Cosine Distance and Jaccard Similarity are introduced but it has some limitations.  WMD is designed to __overcome synonym problem__.The typical example is - Sentence 1: Obama speaks to the media in Illinois- Sentence 2: The president greets the press in ChicagoExcept the stop words, there is no common words among two sentences but both of them are taking about same topic (at that time).WMD use word embeddings to calculate the distance so that it can calculate even though there is no common word. The assumption is that similar words should have similar vectors.First of all, lower case and removing stopwords is an essential step to reduce complexity and preventing misleading. - Sentence 1: obama speaks media illinois- Sentence 2: president greets press chicagoRetrieve vectors from any pre-trained word embeddings models. It can be GloVe, word2vec, fasttext or custom vectors. After that it using normalized bag-of-words (nBOW) to represent the weight or importance. It assumes that higher frequency implies that it is more important.It allows transfer every word from sentence 1 to sentence 2 because algorithm does not know "obama" should transfer to "president". At the end it will choose the minimum transportation cost to transport every word from sentence 1 to sentence 2.  WMD ImplementationBy using gensim, we only need to provide two list of tokens then it will take the rest of calculation<jupyter_code>"""
    News headline get from 
    
    https://www.reuters.com/article/us-musk-tunnel/elon-musks-boring-co-to-build-high-speed-airport-link-in-chicago-idUSKBN1JA224
    http://money.cnn.com/2018/06/14/technology/elon-musk-boring-company-chicago/index.html
    https://www.theverge.com/2018/6/13/17462496/elon-musk-boring-company-approved-tunnel-chicago

"""

news_headline1 = "Elon Musk's Boring Co to build high-speed airport link in Chicago"
news_headline2 = "Elon Musk's Boring Company to build high-speed Chicago airport link"
news_headline3 = "Elon Musk’s Boring Company approved to build high-speed transit between downtown Chicago and O’Hare Airport"
news_headline4 = "Both apple and orange are fruit"

news_headlines = [news_headline1, news_headline2, news_headline3, news_headline4]
# Load Word Embedding Model
import gensim
from gensim.models.keyedvectors import KeyedVectors

print('gensim version: %s' % gensim.__version__)
#glove_model = gensim.models.KeyedVectors.load_word2vec_format('../model/text/stanford/glove/glove.6B.50d.vec')
glove_model = KeyedVectors.load_word2vec_format('/home/fedricio/Desktop/Embeddings_Utilizados/Glove/glove.6B.50d.txt', binary=False, no_header=True)
# Remove stopwords
import spacy
spacy_nlp = spacy.load('en_core_web_lg')

headline_tokens = []
for news_headline in news_headlines:
    headline_tokens.append([token.text.lower() for token in spacy_nlp(news_headline) if not token.is_stop])

print(headline_tokens)
subject_headline = news_headlines[0]
subject_token = headline_tokens[0]

print('Headline: ', subject_headline)
print('=' * 50)
print()

for token, headline in zip(headline_tokens, news_headlines):
    print('-' * 50)
    print('Comparing to:', headline)
    distance = glove_model.wmdistance(subject_token, token)
    print('distance = %.4f' % distance)<jupyter_output>Headline:  Elon Musk's Boring Co to build high-speed airport link in Chicago
==================================================

--------------------------------------------------
Comparing to: Elon Musk's Boring Co to build high-speed airport link in Chicago
distance = 0.0000
--------------------------------------------------
Comparing to: <NAME>k's Boring Company to build high-speed Chicago airport link
distance = 0.0734
--------------------------------------------------
Comparing to: <NAME>k’s Boring Company approved to build high-speed transit between downtown Chicago and O’Hare Airport
distance = 0.3675
--------------------------------------------------
Comparing to: Both apple and orange are fruit
distance = 1.1590<jupyter_text>In gensim implementation, OOV will be removed so that it will not throw an exception or using random vector.  Ejemplo 2 Cosine Similarity & TF IDF<jupyter_code>from string import punctuation
from nltk.corpus import stopwords
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import TfidfVectorizer

language_stopwords = stopwords.words('english')
non_words = list(punctuation)

def remove_stop_words(dirty_text):
    cleaned_text = ''
    for word in dirty_text.split():
        if word in language_stopwords or word in non_words:
            continue
        else:
            cleaned_text += word + ' '
    return cleaned_text

def remove_punctuation(dirty_string):
    for word in non_words:
        dirty_string = dirty_string.replace(word, '')
    return dirty_string

def process_file(file_name):
    file_content = open(file_name, "r").read()
    # All to lower case
    file_content = file_content.lower()
    # Remove punctuation and spanish stopwords
    file_content = remove_punctuation(file_content)
    file_content = remove_stop_words(file_content)
    return file_content

nlp_article = process_file("Archivos 1-General/Ejemplo2/nlp.txt")
sentiment_analysis_article = process_file("Archivos 1-General/Ejemplo2/sentiment_analysis.txt")
java_certification_article = process_file("Archivos 1-General/Ejemplo2/java_cert.txt")

#TF-IDF
vectorizer = TfidfVectorizer ()
X = vectorizer.fit_transform([nlp_article,sentiment_analysis_article,java_certification_article])
#X = count.fit_transform([nlp_article,sentiment_analysis_article,java_certification_article])
similarity_matrix = cosine_similarity(X,X)

print('----------------------------------')
print('Leantechblog article similarity:')
print('----------------------------------')
print(similarity_matrix)

michelle_speech = process_file("Archivos 1-General/Ejemplo2/michelle_speech.txt")
melania_speech = process_file("Archivos 1-General/Ejemplo2/melania_speech.txt")

#TF-IDF
vectorizer = TfidfVectorizer ()
X = vectorizer.fit_transform([michelle_speech,melania_speech])
similarity_matrix = cosine_similarity(X,X)

print('-----------------------------------------')
print('Melania and Michelle speeches similarity:')
print('-----------------------------------------')
print(similarity_matrix)<jupyter_output>----------------------------------
Leantechblog article similarity:
----------------------------------
[[1.         0.217227   0.05744137]
 [0.217227   1.         0.04773379]
 [0.05744137 0.04773379 1.        ]]
-----------------------------------------
Melania and Michelle speeches similarity:
-----------------------------------------
[[1.         0.29814417]
 [0.29814417 1.        ]]<jupyter_text>Ejemplo 3 WMD using Spacy<jupyter_code>#Para desto descargue spacy: >conda install -c conda-forge spacy
#Y el modelo: python -m spacy download en_core_web_sm

import spacy
spacy_nlp = spacy.load('en_core_web_lg')
text = "Some hotel description"
doc = spacy_nlp(text)
current_tokens = [token.text for token in doc]
#
#for item in doc:
#   if item.ent_type_ == "the_type_to_be_removed":
     # remove word from `current_tokens` list
new_text = " ".join(current_tokens)
doc = spacy_nlp(new_text)

#Descargue wdm: pip install wmd
import wmd
spacy_nlp.add_pipe(wmd.WMD.SpacySimilarityHook(spacy_nlp), last=True)
doc_2 = spacy_nlp("Another hotel description")
print(doc.similarity(doc_2))<jupyter_output>0.9536311618244545<jupyter_text>Ejemplo 4 WMD BIEN completo<jupyter_code>from gensim.models import KeyedVectors
import matplotlib.pyplot as  plt
from collections import Counter
import pandas as pd
import numpy as np
import random
import string
import math
import re
import nltk
nltk.download("stopwords")
from nltk.corpus import stopwords<jupyter_output>[nltk_data] Downloading package stopwords to
[nltk_data]     /home/fedricio/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>Usamos el Word2vec descargado<jupyter_code>#Link del cual descargamos el archivo: https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz
EMBEDDING_FILE = '/home/fedricio/Desktop/Embeddings_Utilizados/Word2vec/GoogleNews-vectors-negative300.bin.gz'
word2vec = KeyedVectors.load_word2vec_format(EMBEDDING_FILE, binary=True)<jupyter_output><empty_output><jupyter_text>Leyendo el dataset<jupyter_code>data = pd.read_csv('Archivos 1-General/Ejemplo4/DocumentSimilarity_Dataset.csv')
data = data.sample(frac=1).reset_index(drop=True)

data.head()<jupyter_output><empty_output><jupyter_text>Explicaciones Cosine SImilarity y WDM  1. Cosine SimilarityCosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them.![Alt Text](https://i.imgur.com/HqKjGoQ.jpg) 2. Word Mover's DistanceWord Mover's Distance (WMD) uses the word embeddings of the words in two texts to measure the minimum distance that the words in one text need to travel in semantic space to reach the words in the other text.The WMD is measured by measuring the minimum euclidean distance between each word in the two documents in word2vec space. if the distance is small then words in the two documents are close to each other.So, If I have the same two sentences:- sentence 1: "Obama speaks to the media in Illinois"- sentence 2: "The president greets the press in Chicago"After removing stopwords, The word mover distance is small as mentioned in the figure.![Alt Text](https://imgur.com/L1QNfPK.jpg)<jupyter_code>WORD = re.compile(r"\w+")

WORD    #VER QUE ES ESTO que lo usan abajo.
def text_to_vector(text):
    '''
        converting the document to a term matrix where all words are listed and beside it the frequency of it.

        -- input:
                    text: the document as string 
        -- output:
                    Term matrix: Each word in the two documents and its frequency 
    '''
    words = WORD.findall(text)
    stopwords = nltk.corpus.stopwords.words('english')
    words = [w for w in words if w not in stopwords]
    return Counter(words)

def get_cosine(doc1, doc2):
    '''
        Get the cosine similarity between two documents.
        Depends on the angle between two non zero vectors which are constructed by each word frequency in the two documents.

        -- input:
                      doc1: the first document as string
                      doc2: the second document as string
        -- output:
                      cosine similarity score

    '''
    vec1 = text_to_vector(doc1)
    vec2 = text_to_vector(doc2)
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])

    sum1 = sum([vec1[x] ** 2 for x in list(vec1.keys())])
    sum2 = sum([vec2[x] ** 2 for x in list(vec2.keys())])
    denominator = math.sqrt(sum1) * math.sqrt(sum2)

    if not denominator:
        return 0.0
    else:
        return float(numerator) / denominator
        
def wordMdistance(doc1,doc2):
  '''
      return the word mover distance between two documents 

      -- input:
                      doc1: the first document as list of words
                      doc2: the second document as list of words
      -- output:
                      Word Mover's Distance score
  '''
  sum_dist = 0
  i = 0
  for word in sent1:
    mindist = 1000.0
    for word2 in sent2:
      try:
        j = np.copy(word2vec.get_vector(word))
        t = np.copy(word2vec.get_vector(word2))
        dista = np.sqrt(sum((j-t)**2))
        if(dista < mindist):
          mindist = dista
      except:
        continue
    sum_dist+=mindist
    i+=1
  return sum_dist/i

def WMD(doc1,doc2):
  '''
      Preprocess the document first and remove english stopwords then call the function that calculates the word mover distance
     
      -- input:
                      doc1: the first document as string
                      doc2: the second document as string
      -- output:
                      Word Mover's Distance score
  '''
  first_doc = doc1.lower().split()
  second_doc = doc2.lower().split()
  stopwords = nltk.corpus.stopwords.words('english')
  first_doc = [w for w in first_doc if w not in stopwords]
  second_doc = [w for w in second_doc if w not in stopwords]
  return (word2vec.wmdistance(second_doc, first_doc))<jupyter_output><empty_output><jupyter_text>Visualizamos nuevamente el DS y aplicamos las funciones anteriores<jupyter_code>data.head()
Article_1 = data['articles'][0]
Article_1
Abstract_1 = data['abstracts'][0]
Abstract_1
Sim_1 = data['similarity'][0]
Sim_1
Article_2 = data['articles'][1]
Abstract_2 = data['abstracts'][1]
Sim_2 = data['similarity'][1]
Sim_2<jupyter_output><empty_output><jupyter_text>Get_cosine de a uno<jupyter_code>round(get_cosine(Article_1,Abstract_1),3)
round(get_cosine(Article_2,Abstract_2),3)<jupyter_output><empty_output><jupyter_text>WDM de a uno<jupyter_code>round(WMD(Article_1,Abstract_1),3)
round(WMD(Article_2,Abstract_2),3)<jupyter_output><empty_output><jupyter_text>Nuevo DF con Get_Cosine y WDM para todo el DF anterior.<jupyter_code>articles = data['articles'].astype('str')
abstracts = data['abstracts'].astype('str')
similarities = data['similarity']

NewDF = pd.DataFrame({"article":articles,"summary":abstracts, "sim":similarities})
New_DF_Acotado = NewDF.iloc[0:11]

New_DF_Acotado['Cosine'] = New_DF_Acotado.apply(lambda row: round(get_cosine(row['article'],row['summary']),3), axis=1)
New_DF_Acotado['WMD'] = New_DF_Acotado.apply(lambda row: round(WMD(row['article'],row['summary']),3), axis=1)

New_DF_Acotado<jupyter_output><ipython-input-51-c44c36e84e3f>:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  New_DF_Acotado['Cosine'] = New_DF_Acotado.apply(lambda row: round(get_cosine(row['article'],row['summary']),3), axis=1)
<ipython-input-51-c44c36e84e3f>:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  New_DF_Acotado['WMD'] = New_DF_Acotado.apply(lambda row: round(WMD(row['article'],row['summary']),3), axis=1)<jupyter_text>Ejemplo 5 - Keyword extraction using TF*IDF Son 11 pasos de ejemplo. En resumen: Document -> Remove stop words -> Find Term Frequency (TF) -> Find Inverse Document Frequency (IDF) -> Find TF*IDF -> Get top N Keywords 1. Importamos los paquetes necesarios.We need to tokenize to create word tokens, itemgetter to sort the dictionary, and math to perform log base e operation.<jupyter_code>from nltk import tokenize
from operator import itemgetter
import math<jupyter_output><empty_output><jupyter_text>2. Declaracion de variables.We will declare a string variable. It will be a placeholder for the sample text document<jupyter_code>doc = 'I am a graduate. I want to learn Python. I like learning Python. Python is easy. Python is interesting. Learning increases thinking. Everyone should invest time in learning'<jupyter_output><empty_output><jupyter_text>3. Removemos las stopwordsStopwords are the frequently occurring words that may not carry significance to our analysis. We can remove the using nltk library<jupyter_code>import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize 
stop_words = set(stopwords.words('english'))<jupyter_output><empty_output><jupyter_text>4. Encontramos el número total de palabras (total_words) en el documento. This will be required while calculating Term Frequency<jupyter_code>total_words = doc.split()
total_word_length = len(total_words)
print(total_word_length)<jupyter_output>28<jupyter_text>5. Encontramos el número total de oraciones (total_sent_len).This will be required while calculating Inverse Document Frequency<jupyter_code>total_sentences = tokenize.sent_tokenize(doc)
total_sent_len = len(total_sentences)
print(total_sent_len)<jupyter_output>7<jupyter_text>6. Calculamos TF para cada palabraWe will begin by calculating the word count for each non-stop words and finally divide each element by the result of step 4<jupyter_code>tf_score = {}
for each_word in total_words:
    each_word = each_word.replace('.','')
    if each_word not in stop_words:
        if each_word in tf_score:
            tf_score[each_word] += 1
        else:
            tf_score[each_word] = 1
            
# Dividing by total_word_length for each dictionary element
tf_score.update((x, y/int(total_word_length)) for x, y in tf_score.items())
print(tf_score)<jupyter_output>{'I': 0.10714285714285714, 'graduate': 0.03571428571428571, 'want': 0.03571428571428571, 'learn': 0.03571428571428571, 'Python': 0.14285714285714285, 'like': 0.03571428571428571, 'learning': 0.07142857142857142, 'easy': 0.03571428571428571, 'interesting': 0.03571428571428571, 'Learning': 0.03571428571428571, 'increases': 0.03571428571428571, 'thinking': 0.03571428571428571, 'Everyone': 0.03571428571428571, 'invest': 0.03571428571428571, 'time': 0.03571428571428571}<jupyter_text>7. Funcion para chequear si la palabra (word) está presente en la lista de oraciones (sentences).This method will be required when calculating IDF<jupyter_code>def check_sent(word, sentences): 
    final = [all([w in x for w in word]) for x in sentences] 
    sent_len = [sentences[i] for i in range(0, len(final)) if final[i]]
    return int(len(sent_len))<jupyter_output><empty_output><jupyter_text>8. Calculamos IDF para cada palabra,We will use the function in step 7 to iterate the non-stop word and store the result for Inverse Document Frequency<jupyter_code>idf_score = {}
for each_word in total_words:
    each_word = each_word.replace('.','')
    if each_word not in stop_words:
        if each_word in idf_score:
            idf_score[each_word] = check_sent(each_word, total_sentences)
        else:
            idf_score[each_word] = 1

# Performing a log and divide
idf_score.update((x, math.log(int(total_sent_len)/y)) for x, y in idf_score.items())

print(idf_score)<jupyter_output>{'I': 0.8472978603872037, 'graduate': 1.9459101490553132, 'want': 1.9459101490553132, 'learn': 1.9459101490553132, 'Python': 0.5596157879354227, 'like': 1.9459101490553132, 'learning': 1.252762968495368, 'easy': 1.9459101490553132, 'interesting': 1.9459101490553132, 'Learning': 1.9459101490553132, 'increases': 1.9459101490553132, 'thinking': 1.9459101490553132, 'Everyone': 1.9459101490553132, 'invest': 1.9459101490553132, 'time': 1.9459101490553132}<jupyter_text>9. Calculamos TF * IDFSince the key of both the dictionary is the same, we can iterate one dictionary to get the keys and multiply the values of both<jupyter_code>tf_idf_score = {key: tf_score[key] * idf_score.get(key, 0) for key in tf_score.keys()}
print(tf_idf_score)<jupyter_output>{'I': 0.09078191361291467, 'graduate': 0.06949679103768976, 'want': 0.06949679103768976, 'learn': 0.06949679103768976, 'Python': 0.07994511256220323, 'like': 0.06949679103768976, 'learning': 0.08948306917824057, 'easy': 0.06949679103768976, 'interesting': 0.06949679103768976, 'Learning': 0.06949679103768976, 'increases': 0.06949679103768976, 'thinking': 0.06949679103768976, 'Everyone': 0.06949679103768976, 'invest': 0.06949679103768976, 'time': 0.06949679103768976}<jupyter_text>10. Creamos una funcion para obtener las N más importantes palabras en el documento.<jupyter_code>def get_top_n(dict_elem, n):
    result = dict(sorted(dict_elem.items(), key = itemgetter(1), reverse = True)[:n]) 
    return result<jupyter_output><empty_output><jupyter_text>11. Como prueba obtenemos las 5 palabras top:<jupyter_code>print(get_top_n(tf_idf_score, 5))<jupyter_output>{'I': 0.09078191361291467, 'learning': 0.08948306917824057, 'Python': 0.07994511256220323, 'graduate': 0.06949679103768976, 'want': 0.06949679103768976}
{'fraction_non_alphanumeric': 0.11746600514516722, 'fraction_numerical': 0.058388460124954064, 'mean_word_length': 4.288872691933917, 'pattern_counts': {'":': 4, '<': 83, '<?xml version=': 0, '>': 88, 'https://': 14, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '84926', 'license': 'Apache-2.0', 'n_tokens_mistral': 7894, 'n_tokens_neox': 6957, 'n_words': 1957, 'path': '1-Tests/2-WMD & Cosine Similarity/1-WMD_And_Cosine_Similarity_General.ipynb', 'repo_name': 'FedericoCalonge/automatic_reading_of_CVs_using_text_similarity', 'size': '21772'}