<jupyter_start><jupyter_text>Quora question pairs - Use transformers for question comparisonData: from kaggle<jupyter_code>!pip install -U sentence-transformers


from sentence_transformers import SentenceTransformer, util

import pandas as pd
import numpy as np

from sklearn.metrics.pairwise import cosine_similarity

import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Prepare data - Quora questions<jupyter_code>!wget https://github.com/ravi-ilango/aicamp-mar-2021/blob/main/lab2_3/train.csv.zip?raw=true -O train.csv.zip

!unzip train.csv.zip

df = pd.read_csv('train.csv')
df.head()
df = df[~df['question1'].isnull() & ~df['question2'].isnull()]<jupyter_output><empty_output><jupyter_text>Explore data<jupyter_code>df['length1'] = df['question1'].apply(lambda s: len(s.split()))
df['length1'].hist()
df['length2'] = df['question2'].apply(lambda s: len(s.split()))
df['length2'].hist()
labels = df.is_duplicate.values

plt.hist(labels)
plt.xlabel('target')
plt.ylabel('count')
plt.title('target distribution')
plt.xticks(np.arange(len(np.unique(labels))));
from wordcloud import WordCloud
train_qs = df.question1.values
cloud = WordCloud(width=1440, height=1080).generate(" ".join(train_qs.astype(str)))
plt.figure(figsize=(20, 15))
plt.imshow(cloud)
plt.axis('off')<jupyter_output><empty_output><jupyter_text>Exercise: Draw the wordcloud for the words in question2 column<jupyter_code><jupyter_output><empty_output><jupyter_text>Get a subset of data for similarity detection<jupyter_code>df_ = pd.concat([df[df.is_duplicate == 0].sample(n=5000, random_state=12),
                df[df.is_duplicate == 1].sample(n=5000, random_state=18)])

labels = df_.is_duplicate.values

plt.hist(labels)
plt.xlabel('target')
plt.ylabel('count')
plt.title('target distribution')
plt.xticks(np.arange(len(np.unique(labels))));<jupyter_output><empty_output><jupyter_text>Using sentence transformer<jupyter_code>model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
# Single list of sentences - Possible tens of thousands of sentences
sentences = ['The cat sits outside',
             'A man is playing guitar',
             'I love pasta',
             'The new movie is awesome',
             'The cat plays in the garden',
             'A woman watches TV',
             'The new movie is so great',
             'Do you like pizza?']

#embeddings = model.encode(sentences)
paraphrases = util.paraphrase_mining(model, sentences)
paraphrases[:10]<jupyter_output><empty_output><jupyter_text>Using sentence transformer to score quora questions<jupyter_code>df_['score'] = df_.apply(lambda row: util.paraphrase_mining(model, [row['question1'], row['question2']]), axis=1)

df_.head()<jupyter_output><empty_output><jupyter_text>Exercise:  replace the  token with the score and produce the accuracy report<jupyter_code>from sklearn.metrics import classification_report

thresh_ = .7
y_pred = df_['score'].apply(lambda x: 1 if <REPLACETHIS> > thresh_ else 0)
print(classification_report(df_['is_duplicate'].values, y_pred))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13537964458804524, 'fraction_numerical': 0.01680129240710824, 'mean_word_length': 6.068493150684931, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '387501', 'license': 'MIT', 'n_tokens_mistral': 1103, 'n_tokens_neox': 1045, 'n_words': 209, 'path': 'part2/lab2_3/Quora_deduplicate.ipynb', 'repo_name': 'yasheshshroff/ODSC2021_NLP_PyTorch', 'size': '3095'}