<jupyter_start><jupyter_text>"Quora Questions Pairs using BERT"> "Task: Identify wether two question have similar context/meaning or not"- toc: true- branch: master- badges: true- comments: true- categories: [fastpages, jupyter]- image: images/some_folder/your_image.png- hide: false- search_exclude: true- metadata_key1: metadata_value1- metadata_key2: metadata_value2  Quora Questions Pairs using BERT : Overview**Task: Identify wether two question have similar context/meaning or not**[kaggle](https://www.kaggle.com/c/quora-question-pairs/overview)I have tried this problem using two different approach1.   Using Naive Bayes Classifier2.   Using BERT Naive Bayes Classifier<jupyter_code>import pandas as pd
import numpy as np
import os
import nltk
from nltk.stem import WordNetLemmatizer, PorterStemmer
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
import re
from nltk.corpus import stopwords
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
path = "/content/drive/MyDrive/Quora Questions NLP"
nltk.download()


#load Data
train = pd.read_csv(path+"/train.csv")
print("Total samples:",len(train))
train.head(10)

print(train.isnull().sum(axis=0))#dropping null values
train.dropna(axis=0,inplace=True)<jupyter_output>id              0
qid1            0
qid2            0
question1       1
question2       2
is_duplicate    0
dtype: int64<jupyter_text>Text Preprocessing*   Remove stop words*   Lemmatize<jupyter_code>#preprocessing
def preprocess(series):
  #remove characters other than alphabets & numerics
  words = re.sub("[^A-Za-z0-9]"," ",series).lower().split()

  #lemmatize words
  lemm = WordNetLemmatizer()
  stpwords = stopwords.words('english')
  lemmitized = [lemm.lemmatize(word) for word in words if word not in stpwords]
  sent = ' '.join(lemmitized)
  return sent

train['question1'] =train['question1'].apply(preprocess)#Apply preprocessing
train['question2'] =train['question2'].apply(preprocess)

def concat(ser):#concatenate Question 1 & Question 2
  print(ser['question1'])
  return 1
train['combine'] = train.apply(lambda ser: ser['question1'] + " " + ser['question2'],axis=1)
train.head(10)<jupyter_output><empty_output><jupyter_text>Convert Words into Vector<jupyter_code>cv = TfidfVectorizer(max_features=50000)#Word to Vectors using Tf-Idf

#Take combine questions data as X
X = cv.fit_transform(train['combine'])
y = np.array(train['is_duplicate'])
print(X.shape)

#Tarin-Test Spilt
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.05)
print(X_train.shape,X_test.shape)

naive_model = MultinomialNB()#Training
naive_model.fit(X_train,y_train)

#Predictions
y_pred_train = naive_model.predict(X_train)
y_pred_test = naive_model.predict(X_test)
 
accuracy_train = sum((y_pred_train == y_train).astype(int))/len(y_train)
accuracy_test = sum((y_pred_test == y_test).astype(int))/len(y_test)
print(accuracy_train,accuracy_test)<jupyter_output>0.7558140140390344 0.7431610190452634<jupyter_text>We got 74% Accuracy which is very bad for binary classification problem  BERTI have used "Semantic Similarity with BERT" code to solve this problem.reference : https://keras.io/examples/nlp/semantic_similarity_with_bert/<jupyter_code>import numpy as np
import pandas as pd
import tensorflow as tf
!pip install transformers==2.11.0
import transformers
max_length = 128  # Maximum length of input sentence to the model.
batch_size = 32
epochs = 2

# Labels in our dataset.
labels = [1,0]
#1 : Non Duplicate
#0 : Duplicate
df = pd.read_csv(path+"/train.csv")<jupyter_output><empty_output><jupyter_text>Preprocessing<jupyter_code>print(df.isnull().sum(axis=0))#Dropiing Null values
df.dropna(axis=0,inplace=True)
#create mask for train-test distribution
mask = np.random.rand(len(df)) < 0.7
train_df = df[mask]
not_train = df[~mask]

#create mask for val-test distribution
mask = np.random.rand(len(not_train)) < 0.5
test_df = not_train[mask]
val_df = not_train[~mask]
val_df.head(5)
# Shape of the data
print(f"Total train samples : {train_df.shape[0]}")
print(f"Total validation samples: {val_df.shape[0]}")
print(f"Total test samples: {test_df.shape[0]}")
print("Train Target Distribution")
print(train_df.is_duplicate.value_counts())

print("Validation Target Distribution")
print(val_df.is_duplicate.value_counts())

y_train = tf.keras.utils.to_categorical(train_df.is_duplicate, num_classes=2)#One hot encodding representation
print(f"y_train.shape:{y_train.shape}")

y_val = tf.keras.utils.to_categorical(val_df.is_duplicate, num_classes=2)
print(f"y_val.shape:{y_val.shape}")

y_test = tf.keras.utils.to_categorical(test_df.is_duplicate, num_classes=2)
print(f"y_test.shape:{y_test.shape}")<jupyter_output>y_train.shape:(282550, 2)
y_val.shape:(60634, 2)
y_test.shape:(61103, 2)<jupyter_text>Custom Data Generator<jupyter_code>class BertSemanticDataGenerator(tf.keras.utils.Sequence):
    """Generates batches of data.

    Args:
        sentence_pairs: Array of premise and hypothesis input sentences.
        labels: Array of labels.
        batch_size: Integer batch size.
        shuffle: boolean, whether to shuffle the data.
        include_targets: boolean, whether to incude the labels.

    Returns:
        Tuples `([input_ids, attention_mask, `token_type_ids], labels)`
        (or just `[input_ids, attention_mask, `token_type_ids]`
         if `include_targets=False`)
    """

    def __init__(
        self,
        sentence_pairs,
        labels,
        batch_size=batch_size,
        shuffle=True,
        include_targets=True,
    ):
        self.sentence_pairs = sentence_pairs
        self.labels = labels
        self.shuffle = shuffle
        self.batch_size = batch_size
        self.include_targets = include_targets
        
        # Load our BERT Tokenizer to encode the text.
        # We will use base-base-uncased pretrained model.
        self.tokenizer = transformers.BertTokenizer.from_pretrained(
            "bert-base-uncased", do_lower_case=True
        )
        self.indexes = np.arange(len(self.sentence_pairs))
        self.on_epoch_end()

    def __len__(self):
        # Denotes the number of batches per epoch.
        return len(self.sentence_pairs) // self.batch_size

    def __getitem__(self, idx):
        # Retrieves the batch of index.
        indexes = self.indexes[idx * self.batch_size : (idx + 1) * self.batch_size]
        sentence_pairs = self.sentence_pairs[indexes]

        # With BERT tokenizer's batch_encode_plus batch of both the sentences are
        # encoded together and separated by [SEP] token.
        encoded = self.tokenizer.batch_encode_plus(
            sentence_pairs.tolist(),
            add_special_tokens=True,
            max_length=max_length,
            return_attention_mask=True,
            return_token_type_ids=True,
            pad_to_max_length=True,
            return_tensors="tf",
        )

        # Convert batch of encoded features to numpy array.
        input_ids = np.array(encoded["input_ids"], dtype="int32")
        attention_masks = np.array(encoded["attention_mask"], dtype="int32")
        token_type_ids = np.array(encoded["token_type_ids"], dtype="int32")

        # Set to true if data generator is used for training/validation.
        if self.include_targets:
            labels = np.array(self.labels[indexes], dtype="int32")
            return [input_ids, attention_masks, token_type_ids], labels
        else:
            return [input_ids, attention_masks, token_type_ids]

    def on_epoch_end(self):
        # Shuffle indexes after each epoch if shuffle is set to True.
        if self.shuffle:
            np.random.RandomState(42).shuffle(self.indexes)<jupyter_output><empty_output><jupyter_text>Build The Model<jupyter_code>strategy = tf.distribute.MirroredStrategy()# Create the model under a distribution strategy scope.

with strategy.scope():
    # Encoded token ids from BERT tokenizer.
    input_ids = tf.keras.layers.Input(
        shape=(max_length,), dtype=tf.int32, name="input_ids"
    )
    # Attention masks indicates to the model which tokens should be attended to.
    attention_masks = tf.keras.layers.Input(
        shape=(max_length,), dtype=tf.int32, name="attention_masks"
    )
    # Token type ids are binary masks identifying different sequences in the model.
    token_type_ids = tf.keras.layers.Input(
        shape=(max_length,), dtype=tf.int32, name="token_type_ids"
    )
    # Loading pretrained BERT model.
    bert_model = transformers.TFBertModel.from_pretrained("bert-base-uncased")
    # Freeze the BERT model to reuse the pretrained features without modifying them.
    bert_model.trainable = False

    sequence_output, pooled_output = bert_model(
        input_ids, attention_mask=attention_masks, token_type_ids=token_type_ids
    )
    # Add trainable layers on top of frozen layers to adapt the pretrained features on the new data.
    bi_lstm = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True))(sequence_output)
    
    # Applying hybrid pooling approach to bi_lstm sequence output.
    avg_pool = tf.keras.layers.GlobalAveragePooling1D()(bi_lstm)
    max_pool = tf.keras.layers.GlobalMaxPooling1D()(bi_lstm)
    concat = tf.keras.layers.concatenate([avg_pool, max_pool])
    dropout = tf.keras.layers.Dropout(0.3)(concat)
    output = tf.keras.layers.Dense(2, activation="softmax")(dropout)
    model = tf.keras.models.Model(
        inputs=[input_ids, attention_masks, token_type_ids], outputs=output
    )

    model.compile(
        optimizer=tf.keras.optimizers.Adam(),
        loss="categorical_crossentropy",
        metrics=["acc"],
    )


print(f"Strategy: {strategy}")
model.summary()<jupyter_output>WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
Strategy: <tensorflow.python.distribute.mirrored_strategy.MirroredStrategy object at 0x7fa42b661090>
Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_ids (InputLayer)          [(None, 128)]        0                                            
__________________________________________________________________________________________________
attention_masks (InputLayer)    [(None, 128)]        0                                            
______________________________________________________________________________________[...]<jupyter_text>Create train and validation data generators<jupyter_code>train_data = BertSemanticDataGenerator(
    train_df[["question1", "question2"]].values.astype("str"),
    y_train,
    batch_size=batch_size,
    shuffle=True,
)
val_data = BertSemanticDataGenerator(
    val_df[["question1", "question2"]].values.astype("str"),
    y_val,
    batch_size=batch_size,
    shuffle=False,
)<jupyter_output><empty_output><jupyter_text>Train the modelTraining is done only for the top layers to perform "feature extraction", which will allow the model to use the representations of the pretrained model.<jupyter_code>history = model.fit(
    train_data,
    validation_data=val_data,
    epochs=epochs,
     use_multiprocessing=True,
    workers=-1,
)<jupyter_output>Epoch 1/2
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
8850/8850 [==============================] - 3281s 366ms/step - loss: 0.4316 - acc: 0.7859 - val_loss: 0.3371 - val_acc: 0.8432
Epoch 2/2
8850/8850 [=[...]<jupyter_text>Fine TuningNow BERT model has knowledge of Language & Context now we can unfreeze the BERT pretrained weights & retrain using very low learning rate to solve actual NLP problem<jupyter_code>bert_model.trainable = True
# Recompile the model to make the change effective.
model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-5),
    loss="categorical_crossentropy",
    metrics=["accuracy"],
)
model.summary()
history = model.fit(
    train_data,
    validation_data=val_data,
    epochs=epochs,
    use_multiprocessing=True,
    workers=-1,
)<jupyter_output>Epoch 1/2
8850/8850 [==============================] - 7739s 874ms/step - loss: 0.2853 - accuracy: 0.8741 - val_loss: 0.2712 - val_accuracy: 0.8832
Epoch 2/2
8850/8850 [==============================] - 7737s 874ms/step - loss: 0.2139 - accuracy: 0.9100 - val_loss: 0.2477 - val_accuracy: 0.8973<jupyter_text>After Waiting of 4-5 hours nowour model is trained!  Evaluate on Test Dataset<jupyter_code>test_data = BertSemanticDataGenerator(
    test_df[["question1", "question2"]].values.astype("str"),
    y_test,
    batch_size=batch_size,
    shuffle=False,
)
model.evaluate(test_data, verbose=1)<jupyter_output>1890/1890 [==============================] - 529s 280ms/step - loss: 0.2438 - accuracy: 0.9001<jupyter_text>We Got 90% Accuracy on Test Dataset which is far better than Naive Bayes<jupyter_code>def check_similarity(sentence1, sentence2):
  sentence_pairs = np.array([[str(sentence1), str(sentence2)]])
  test_data = BertSemanticDataGenerator(
      sentence_pairs, labels=None, batch_size=1, shuffle=False, include_targets=False,
  )

  proba = model.predict(test_data)[0]
  idx = np.argmax(proba)
  proba = f"{proba[idx]: .2f}%"
  pred = labels[idx]
  return pred, proba<jupyter_output><empty_output><jupyter_text>Try the custom Questions<jupyter_code>ind = np.random.randint(0,500) 
#Duplicate Questions
q1 = test_df[test_df["is_duplicate"] == 1].iloc[ind]['question1']
q2 = test_df[test_df["is_duplicate"] == 1].iloc[ind]['question2']
print(q1+"\n"+q2)
check_similarity(q1,q2)
ind = np.random.randint(0,500)
#Non-Duplicate Questions
q1 = test_df[test_df["is_duplicate"] == 0].iloc[ind]['question1']
q2 = test_df[test_df["is_duplicate"] == 0].iloc[ind]['question2']
print(q1+"\n"+q2)
check_similarity(q1,q2)<jupyter_output>How many pullups can an average person do?
How many photos a day does an average person take with their phone.?
{'fraction_non_alphanumeric': 0.12505023442732752, 'fraction_numerical': 0.027059611520428666, 'mean_word_length': 4.491357116586981, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 47, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '251134', 'license': 'Apache-2.0', 'n_tokens_mistral': 5169, 'n_tokens_neox': 4880, 'n_words': 1042, 'path': '_notebooks/2021-04-18-Quora Questions Pairs.ipynb', 'repo_name': 'karm216/Finger-Count-ResNet', 'size': '14930'}