<jupyter_start><jupyter_text>Intent Detection : Sum of Word2Vec The purpose of this python notebook is to study the use of word vector representations to categorize sentences.  My idea is to sum the vectors of every word of a sentence, and then compute the similarity between 2 sentences in order to test if they belong to the same category. Firstly, in order to use vector representations of words, we need to import the [gensim library](https://radimrehurek.com/gensim/index.html)<jupyter_code>import gensim<jupyter_output><empty_output><jupyter_text>Instead of training models on huge data sets, we will use a a pre-trained vectors trained on part of Google News dataset (about 100 billion words). The model contains 300-dimensional vectors for 3 million words and phrases. The phrases were obtained using a simple data-driven approach described in (<NAME>, <NAME>, <NAME>, <NAME>, and <NAME>. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.).  The archive is available here: [GoogleNews-vectors-negative300.bin.gz](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing).<jupyter_code>model = gensim.models.Word2Vec.load_word2vec_format("GoogleNews-vectors-negative300.bin", binary=True)<jupyter_output><empty_output><jupyter_text>To verify if all is ok, just compute the similarity between words 'woman' and 'man'<jupyter_code>model.similarity('woman', 'man')<jupyter_output><empty_output><jupyter_text>To get the similarity between 2 vectors, we need a function compute the cosine similarity (Cf. https://en.wikipedia.org/wiki/Cosine_similarity).<jupyter_code>import math

def cosine_similarity(v1,v2):
    "compute cosine similarity of v1 to v2: (v1 dot v2)/(||v1||*||v2||)"
    sumxx, sumxy, sumyy = 0, 0, 0
    for i in range(len(v1)):
        x = v1[i]
        y = v2[i]
        sumxx += x*x
        sumyy += y*y
        sumxy += x*y
    return sumxy/math.sqrt(sumxx*sumyy)<jupyter_output><empty_output><jupyter_text>Let's test this function in comparison with model.similarity():<jupyter_code>model.similarity('woman', 'man') - cosine_similarity(model['woman'], model['man'])<jupyter_output><empty_output><jupyter_text>The results are not exactly the same, but for now, I don't have time to find out the reason why... Then, let's compare 3 sentences in order to find out if they belong to the same category: 1. I'm starving2. I want to eat pizza3. What are the news today?First, we need to sum the vectors of each words to get a vector representing each sentence.<jupyter_code>sentence1 = model['I']+model['m']+model['starving']<jupyter_output><empty_output><jupyter_text>*Note:* 'to' is a stopword, so it has to be removed.<jupyter_code>sentence2 = model['I']+model['want']+model['eat']+model['pizza']<jupyter_output><empty_output><jupyter_text>Then, let's compute the similarity between "I'm starving" and "I want to eat pizza".<jupyter_code>cosine_similarity(sentence1, sentence2)<jupyter_output><empty_output><jupyter_text>One last sentence:<jupyter_code>sentence3 = model['What']+model['are']+model['the']+model['news']+model['today']<jupyter_output><empty_output><jupyter_text>Compute the similarity between the last sentence and the two previous ones:<jupyter_code>cosine_similarity(sentence1, sentence3)
cosine_similarity(sentence2, sentence3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1081802015411974, 'fraction_numerical': 0.016004742145820983, 'mean_word_length': 6.305194805194805, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '576050', 'license': 'MIT', 'n_tokens_mistral': 1128, 'n_tokens_neox': 1053, 'n_words': 331, 'path': 'sum_word2vec.ipynb', 'repo_name': 'SebastienBoisard/IntentDetection', 'size': '3403'}