<jupyter_start><jupyter_text>Quora Question Pairs  ObjectivesIdentify which questions asked on Quora are duplicates of questions that have already been asked. This could be useful to instantly provide answers to questions that have already been answered. The task is to predict whether a pair of questions are duplicates or not.  **Credits**: Kaggle (https://www.kaggle.com/c/quora-question-pairs) It is a binary classification problem, for a given pair of questions we need to predict if they are duplicate or not.  Performance MetricsPredictions are evaluated on the following metrics between the predicted values and the ground truth.- Log Loss (https://www.kaggle.com/wiki/LogarithmicLoss)- Binary Confusion Matrix<jupyter_code># importing the libraries

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

import re
import pickle
import string
import unidecode

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem.porter import PorterStemmer

from os import path
from wordcloud import WordCloud, STOPWORDS

from fuzzywuzzy import fuzz
from fuzzywuzzy import process
# importing the dataset

df = pd.read_csv('questions.csv')
df.head(10)
df.info()
df.describe()<jupyter_output><empty_output><jupyter_text>We are given a minimal number of data fields here, consisting of:- id:  A simple row ID- qid1, qid2:  The unique ID of each question in the pair- question1, question2:  The actual textual contents of the questions.- is_duplicate:  The label that we are trying to predict - whether the two questions are duplicates of each other.<jupyter_code># drop ID fields

df.drop(['id', 'qid1', 'qid2'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Exploratory AnalysisDistribution of data points<jupyter_code>df['is_duplicate'].value_counts()
df['is_duplicate'].value_counts().plot.bar()
print("Total number of questions: {}".format(df.shape[0]))
# number of duplicate and non-duplicate questions

print("Question pairs that are not similar (is_duplicate=0): {}".format(
    round((df['is_duplicate'] == 0).sum() * 100 / df.shape[0], 2)))

print("Question pairs that are similar (is_duplicate=1): {}".format(
round((df['is_duplicate'] == 1).sum() * 100 / df.shape[0], 2)))
round((df['is_duplicate'] == 0).sum() / df.shape[0], 2)
# distribution of questions

questions = pd.concat([df['question1'], df['question2']])
questions_uniq = questions.unique()
questions_more_than_one = (questions.value_counts() > 1).sum()

print("Total number of questions: {}".format(questions.shape[0]))
print("Total number of unique questions: {}".format(questions_uniq.shape[0]))
print("Total number of duplicate questions occuring more than once: {} %".format(
    round(questions_more_than_one / questions_uniq.shape[0] * 100, 2)))
print("Maximum occurence of single repeated question: {}".format(questions.value_counts().max()))
x = ['Unique questions', 'Repeated questions']
y = [questions_uniq.shape[0], questions_more_than_one]
plt.figure(figsize=(8, 6))
plt.title('Distribution of unique and repeated questions')
sns.barplot(x, y)
plt.show()
# histogram of questions frequency

plt.figure(figsize=(20, 10))
plt.yscale('log', nonposy='clip')
plt.title('Log-Histogram of Questions Frequency')
plt.xlabel('Number of occurences of question')
plt.ylabel('Number of questions')
plt.hist(questions.value_counts(), bins=160)
plt.show()
# check missing values

df.isnull().sum()
# inspecting the empty rows

empty_rows = df[df.isnull().any(1)]
empty_rows
# deleting empty rows

df.drop(empty_rows.index, inplace=True)<jupyter_output><empty_output><jupyter_text>Feature extraction- Number of characters in question1 and question2- Number of words in question1 and question2- Number of characters in question1 and question2 (removing whitespaces)- Difference between number of characters in question1 and question2- Difference between number of words in question1 and question2- Difference between number of characters in question1 and question2 (removing whitespaces)- Number of common words in question1 and question2- Common words ratio i.e. Number of common words in question1 and question2 / Total number of words in question1 and question2<jupyter_code># extracting features from data

df['q1_len'] = df['question1'].apply(lambda x: len(x))
df['q2_len'] = df['question2'].apply(lambda x: len(x))
df['q1_word_len'] = df['question1'].apply(lambda x: len(x.split(' ')))
df['q2_word_len'] = df['question2'].apply(lambda x: len(x.split(' ')))
df['q1_char_len'] = df['question1'].apply(lambda x: len(x.replace(' ', '')))
df['q2_char_len'] = df['question2'].apply(lambda x: len(x.replace(' ', '')))
df['len_diff'] = df.apply(lambda x: abs(x['q1_len'] - x['q2_len']), axis=1)
df['word_len_diff'] = df.apply(lambda x: abs(x['q1_word_len'] - x['q2_word_len']), axis=1)
df['char_len_diff'] = df.apply(lambda x: abs(x['q1_char_len'] - x['q2_char_len']), axis=1)
df['common_words'] = df.apply(lambda x: len(
    set(x['question1'].lower().split(' ')) & set(x['question2'].lower().split(' '))), axis=1)
df['common_words_ratio'] = df.apply(lambda x: x['common_words'] / (x['q1_word_len'] + x['q2_word_len']), axis=1)
df.iloc[:, 2:].head()
print("Minimum length of the questions in question 1: {}".format(min(df['q1_len'])))
print("Minimum length of the questions in question 2: {}".format(min(df['q2_len'])))

print("Number of questions with minimum length in question 1: {}".format((df['q1_len'] == 1).sum()))
print("Number of questions with minimum length in question 2: {}".format((df['q2_len'] == 1).sum()))
plt.figure(figsize=(10, 8))
plt.title('Distribution of common words ratio')
sns.distplot(df[df['is_duplicate'] == 1]['common_words_ratio'], label = "1", color = 'red')
sns.distplot(df[df['is_duplicate'] == 0]['common_words_ratio'], label = "0", color = 'blue')
plt.show()<jupyter_output><empty_output><jupyter_text>Text Preprocessing- Remove HTML tags- Remove extra whitespaces- Convert accented characters to ASCII characters- Expand contractions- Remove special characters- Lowercase all texts- Remove stopwords- Lemmatization<jupyter_code># remove html tags from text

def remove_html_tags(text):
    """Remove html tags from a string"""
    clean = re.compile('<.*?>')
    return re.sub(clean, '', text)

remove_html_tags("<p>This is a paragraph</p>")
# words with accent marks like 'latté' and 'café' can be converted and standardized to just 'latte' and 'cafe'

def remove_accented_chars(text):
    """remove accented characters from text, e.g. café"""
    text = unidecode.unidecode(text)
    return text

remove_accented_chars('latté')
# expand contractions

cList = {
    "ain't": "am not",
    "aren't": "are not",
    "can't": "cannot",
    "can't've": "cannot have",
    "'cause": "because",
    "could've": "could have",
    "couldn't": "could not",
    "couldn't've": "could not have",
    "didn't": "did not",
    "doesn't": "does not",
    "don't": "do not",
    "hadn't": "had not",
    "hadn't've": "had not have",
    "hasn't": "has not",
    "haven't": "have not",
    "he'd": "he would",
    "he'd've": "he would have",
    "he'll": "he will",
    "he'll've": "he will have",
    "he's": "he is",
    "how'd": "how did",
    "how'd'y": "how do you",
    "how'll": "how will",
    "how's": "how is",
    "I'd": "I would",
    "I'd've": "I would have",
    "I'll": "I will",
    "I'll've": "I will have",
    "I'm": "I am",
    "I've": "I have",
    "isn't": "is not",
    "it'd": "it had",
    "it'd've": "it would have",
    "it'll": "it will",
    "it'll've": "it will have",
    "it's": "it is",
    "let's": "let us",
    "ma'am": "madam",
    "mayn't": "may not",
    "might've": "might have",
    "mightn't": "might not",
    "mightn't've": "might not have",
    "must've": "must have",
    "mustn't": "must not",
    "mustn't've": "must not have",
    "needn't": "need not",
    "needn't've": "need not have",
    "o'clock": "of the clock",
    "oughtn't": "ought not",
    "oughtn't've": "ought not have",
    "shan't": "shall not",
    "sha'n't": "shall not",
    "shan't've": "shall not have",
    "she'd": "she would",
    "she'd've": "she would have",
    "she'll": "she will",
    "she'll've": "she will have",
    "she's": "she is",
    "should've": "should have",
    "shouldn't": "should not",
    "shouldn't've": "should not have",
    "so've": "so have",
    "so's": "so is",
    "that'd": "that would",
    "that'd've": "that would have",
    "that's": "that is",
    "there'd": "there had",
    "there'd've": "there would have",
    "there's": "there is",
    "they'd": "they would",
    "they'd've": "they would have",
    "they'll": "they will",
    "they'll've": "they will have",
    "they're": "they are",
    "they've": "they have",
    "to've": "to have",
    "wasn't": "was not",
    "we'd": "we had",
    "we'd've": "we would have",
    "we'll": "we will",
    "we'll've": "we will have",
    "we're": "we are",
    "we've": "we have",
    "weren't": "were not",
    "what'll": "what will",
    "what'll've": "what will have",
    "what're": "what are",
    "what's": "what is",
    "what've": "what have",
    "when's": "when is",
    "when've": "when have",
    "where'd": "where did",
    "where's": "where is",
    "where've": "where have",
    "who'll": "who will",
    "who'll've": "who will have",
    "who's": "who is",
    "who've": "who have",
    "why's": "why is",
    "why've": "why have",
    "will've": "will have",
    "won't": "will not",
    "won't've": "will not have",
    "would've": "would have",
    "wouldn't": "would not",
    "wouldn't've": "would not have",
    "y'all": "you all",
    "y'alls": "you alls",
    "y'all'd": "you all would",
    "y'all'd've": "you all would have",
    "y'all're": "you all are",
    "y'all've": "you all have",
    "you'd": "you had",
    "you'd've": "you would have",
    "you'll": "you you will",
    "you'll've": "you you will have",
    "you're": "you are",
    "you've": "you have"
}

c_re = re.compile('(%s)' % '|'.join(cList.keys()))

def expand_contractions(text, c_re=c_re):
    def replace(match):
        return cList[match.group(0)]
    return c_re.sub(replace, text)

print(expand_contractions('I ain\'t got time for y\'alls foolishness'))
print(expand_contractions('You won\'t live to see tomorrow.'))
# remove comma between numbers

def remove_comma_between_numbers(text):
    return re.sub('(?<=[0-9])\,(?=[0-9])', "", text)

print(remove_comma_between_numbers("15,000"))
print(remove_comma_between_numbers("lorem ipsum 1,00,000"))
# load stop words and stemmer

stop_words = set(stopwords.words('english')) 
porter = PorterStemmer()
def clean(text, remove_html=True, remove_punctuation=True):

    text = text.lower()

    if remove_html:
        text = remove_html_tags(text)

    text = remove_accented_chars(text)
    text = expand_contractions(text)
    text = remove_comma_between_numbers(text)

    # remove punctuation (optional)
    if remove_punctuation:
        text = text.translate(str.maketrans('', '', string.punctuation))

    # remove stop words
    text_tokens = word_tokenize(text)
    filtered_text = [w for w in text_tokens if not w in stop_words]

    # stemming using porter
    stemmed_text = [porter.stem(w) for w in filtered_text]
    text = ' '.join(filtered_text)

    return text
df['question1'] = df['question1'].apply(clean)
df['question2'] = df['question2'].apply(clean)
for i in range(10):
    print(df.question1[i])
    print(df.question2[i])
    print('')
df_duplicate = df[df['is_duplicate'] == 1]
df_non_duplicate = df[df['is_duplicate'] == 0]
# merge question1 and question2 in duplicate and non_duplocate dataset

p = np.dstack([df_duplicate["question1"], df_duplicate["question2"]]).flatten()
n = np.dstack([df_non_duplicate["question1"], df_non_duplicate["question2"]]).flatten()

print("Number of data points in class 1 (duplicate pairs): ", len(p))
print("Number of data points in class 0 (non duplicate pairs): ", len(n))

# saving the np array into a text file
np.savetxt('train_p.txt', p, delimiter=' ', fmt='%s')
np.savetxt('train_n.txt', n, delimiter=' ', fmt='%s')
d = path.dirname('.')

textp_w = open(path.join(d, 'train_p.txt')).read()
textn_w = open(path.join(d, 'train_n.txt')).read()
stopwords = set(STOPWORDS)

print ("Total number of words in duplicate pair questions :",len(textp_w))
print ("Total number of words in non duplicate pair questions :",len(textn_w))
# word cloud for duplicate questions pair

plt.figure(figsize=(12, 8))
wc = WordCloud(background_color="white", max_words=len(textp_w), stopwords=stopwords)
wc.generate(textp_w)
plt.title("Word Cloud for duplicate question pairs")
plt.imshow(wc, interpolation='bilinear')
plt.axis("off")
plt.show()
# word cloud for non duplicate questions pair

plt.figure(figsize=(12, 8))
wc = WordCloud(background_color="white", max_words=len(textn_w), stopwords=stopwords)
wc.generate(textn_w)
plt.title("Word Cloud for non duplicate question pairs")
plt.imshow(wc, interpolation='bilinear')
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>Advanced Feature Extraction  FuzzyWuzzyThe following text ratio can be extracted from the fuzzywuzzy library.- Fuzz Ratio- Fuzz Parial Ratio- Token Sort Ratio- Token Set Ratio**Credits:**https://github.com/seatgeek/fuzzywuzzyhttp://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/<jupyter_code>df['fuzz_ratio'] = df.apply(lambda x: fuzz.ratio(x['question1'], x['question2']), axis=1)
df['fuzz_partial_ratio'] = df.apply(lambda x: fuzz.partial_ratio(x['question1'], x['question2']), axis=1)
df['token_sort_ratio'] = df.apply(lambda x: fuzz.token_sort_ratio(x['question1'], x['question2']), axis=1)
df['token_set_ratio'] = df.apply(lambda x: fuzz.token_set_ratio(x['question1'], x['question2']), axis=1)
plt.figure(figsize=(10, 8))
plt.title('Distribution of fuzz_ratio')
sns.distplot(df[df['is_duplicate'] == 1]['fuzz_ratio'], label = "1", color = 'red')
sns.distplot(df[df['is_duplicate'] == 0]['fuzz_ratio'], label = "0", color = 'blue')
plt.show()
plt.figure(figsize=(10, 8))
plt.title('Distribution of fuzz_partial_ratio')
sns.distplot(df[df['is_duplicate'] == 1]['fuzz_partial_ratio'], label = "1", color = 'red')
sns.distplot(df[df['is_duplicate'] == 0]['fuzz_partial_ratio'], label = "0", color = 'blue')
plt.show()
plt.figure(figsize=(10, 8))
plt.title('Distribution of token_sort_ratio')
sns.distplot(df[df['is_duplicate'] == 1]['token_sort_ratio'], label = "1", color = 'red')
sns.distplot(df[df['is_duplicate'] == 0]['token_sort_ratio'], label = "0", color = 'blue')
plt.show()
plt.figure(figsize=(10, 8))
plt.title('Distribution of token_set_ratio')
sns.distplot(df[df['is_duplicate'] == 1]['token_set_ratio'], label = "1", color = 'red')
sns.distplot(df[df['is_duplicate'] == 0]['token_set_ratio'], label = "0", color = 'blue')
plt.show()
df.head(10)
# saving the data

df.to_pickle("./questions_preprocessed.pkl")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17618347151644825, 'fraction_numerical': 0.012904519925113666, 'mean_word_length': 5.03591606133979, 'pattern_counts': {'":': 118, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 3, 'lorem ipsum': 1, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '61571', 'license': 'MIT', 'n_tokens_mistral': 5484, 'n_tokens_neox': 5118, 'n_words': 1415, 'path': 'analysis_and_preprocessing.ipynb', 'repo_name': 'maanavshah/quora-question-pairs', 'size': '14956'}