<jupyter_start><jupyter_text>[Word2vec](https://code.google.com/archive/p/word2vec/) modelDownload here : https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit<jupyter_code># Loading the model with gensim
from gensim.models import KeyedVectors
import pandas as pd
import os

MODEL_PATH = '/home/b/Downloads/GoogleNews-vectors-negative300.bin.gz'

if not os.path.exists(MODEL_PATH):
    raise ValueError("SKIP: You need to download the google news model")
    
model = KeyedVectors.load_word2vec_format(MODEL_PATH, binary=True,limit=200000)
from nltk.corpus import stopwords
from nltk import wordpunct_tokenize

plus = {'hdfc'}
minus = {'what','how','where','when','why'}
stop = set(stopwords.words('english'))

stop.update(plus)
stop.difference_update(minus)

def get_tokens(sent):
    return [x for x in wordpunct_tokenize(sent.lower()) if x.isalnum() and x not in stop]
sentence_obama = 'Obama speaks to the media in Illinois'
sentence_obama_past = 'In Illinois, Obama spoke to journalists'
sentence_president = 'The president greets the press in Chicago'
sentence_arbitrary = 'Mary had a little lamb'

tokens_obama = get_tokens(sentence_obama)
tokens_president = get_tokens(sentence_president)
tokens_obama_past = get_tokens(sentence_obama_past)
tokens_arbitrary = get_tokens(sentence_arbitrary)

print ("'"+sentence_obama+"' vs. '"+sentence_obama+"' \t\t:",model.wmdistance(tokens_obama, tokens_obama))
print ("'"+sentence_obama+"' vs. '"+sentence_obama_past+"' \t\t:",model.wmdistance(tokens_obama, tokens_obama_past))
print ("'"+sentence_obama+"' vs. '"+sentence_president+"' \t:",model.wmdistance(tokens_obama, tokens_president))
print ("'"+sentence_obama+"' vs. '"+sentence_arbitrary+"' \t\t\t\t:",model.wmdistance(tokens_obama, tokens_arbitrary))<jupyter_output>'Obama speaks to the media in Illinois' vs. 'Obama speaks to the media in Illinois' 		: 0.0
'Obama speaks to the media in Illinois' vs. 'In Illinois, Obama spoke to journalists' 		: 1.751374609270841
'Obama speaks to the media in Illinois' vs. 'The president greets the press in Chicago' 	: 3.3901614766891077
'Obama speaks to the media in Illinois' vs. '<NAME> a little lamb' 				: 4.176404432301822<jupyter_text>Loading the dataset<jupyter_code>df = pd.read_excel('hdfc.xlsx')
df = df.drop_duplicates('Question')
df = df.reset_index()

qlabels = df['Question'].to_dict()

drop_list = []
for x in qlabels:
    for w in get_tokens(qlabels[x]):
        if w not in model.vocab:
            drop_list.append(x)

reduced = df.drop(drop_list)

qlabels = reduced['Question'].to_dict()
alabels = reduced['Answer'].to_dict()
reduced<jupyter_output><empty_output><jupyter_text>Finding closest question using WmdSimilarity<jupyter_code>from gensim.similarities import WmdSimilarity
questions = reduced['Question'].values

instance = WmdSimilarity([get_tokens(x) for x in questions], model, num_best=5)
def predict_wmd(sent):
    query = get_tokens(sent)
    sims = instance[query]
    scores = []
    for i in range(5):
        scores.append({"Score":'%.4f' % sims[i][1],"Question":questions[sims[i][0]]})
    return pd.DataFrame(scores)
predict('What is the procedure to pay my Personal Loan?')<jupyter_output><empty_output><jupyter_text>WMD Similarity vs. TF-IDF- WMD     - Uses word2vec to embed and commpare words.     - Hence distance between two words (which is used to compute distance between documents) reflects their real-life relationship    - Computationally expensive. `O(p^3*logp)` , where p is the size of the corpus- TF-IDF     - Doesn't reflect real-life relationships    - Importance/weight assigned to each word is inversely proportional to its frequency in the corpus    - Faster to compute than wmd since vocabulary is much smaller    - Cannot matches questions with synonyms<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
from nltk import FreqDist
import numpy as np

# Training tfidf vectoriser
tf_vect =TfidfVectorizer(stop_words=stop,
                         lowercase=True,
                         use_idf=True)
all_qs_vectors = tf_vect.fit_transform(questions)

all_words = wordpunct_tokenize(' '.join(questions).lower())
fdist = FreqDist(all_words)

# Function to predict closest question using TF-IDF
def predict(query,n=5,answers=False):
    query_vector = tf_vect.transform([query])
    arr = np.dot(all_qs_vectors, query_vector.T).toarray().flatten()
    matches,results = arr.argsort(axis=0)[::-1],[]
    for i in matches[:n]:
        res = {"Question":questions[i],"Score":arr[i]}
        results.append(res)
    return pd.DataFrame(results)
query = 'What is the procedure to pay my Personal Loan?'
word_dist = [{"word":w,"count":fdist[w]} for w in get_tokens(query)]
pd.DataFrame(word_dist).sort_values('count')<jupyter_output><empty_output><jupyter_text>Consider the question : **What is the procedure to pay my Personal Loan?**- TFIDF suggests '*What is the cancellation procedure*' by matching the words **procedure** and **what**    - This is closely followed by '*How can I repay my Personal Loan?*' by matching the words **Personal** and **Loan**    - Thus, the word **procedure** is assigned greater importance simply due to lack of questions with **procedure**    - This worked to our benefit in the previous notebook where **Amortization** was assigned greater importance but backfires when common words aren't just as common/frequent in the corpus    - Hence, WMD performs a lot better than tfidf when we have a limited set of questions- WMD suggests '*How can I repay my Personal Loan?*' by roughly matching    - **What is the procedure** to **How**    - **pay** to **repay**<jupyter_code>predict('What is the procedure to pay my Personal Loan?')
predict_wmd('What is the procedure to pay my Personal Loan?')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1144141030072589, 'fraction_numerical': 0.013480815762184583, 'mean_word_length': 5.189304812834225, 'pattern_counts': {'":': 6, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '382725', 'license': 'MIT', 'n_tokens_mistral': 1969, 'n_tokens_neox': 1834, 'n_words': 542, 'path': '.ipynb_checkpoints/wmd_exp2-checkpoint.ipynb', 'repo_name': 'py-ranoid/FAQ-Bot-Notebooks', 'size': '5788'}