<jupyter_start><jupyter_text>NLP with disaster tweets!(BERT explained) (~84.5% Accuracy)About the problemIn this competition, you’re challenged to build a machine learning model that predicts which Tweets are about real disasters and which one’s aren’t.I have two notebooks on this competition , the first one is using basic naive-base model whereas this one is by using BERT pre-trained model. If you're a beginner I highly recommend you to check out the basic model notebook first ! Here is the link  [NLP w/ Disaster tweets!(Explained)](https://www.kaggle.com/friskycodeur/nlp-w-disaster-tweets-explained) Here are the things I will try to cover in this Notebook:- Defining helper functions- Data cleaning (basic and advanced both)- Using BERT model to get higher accuracy- Is it posible to get 100% accuracy and if yes then how can you get that ! If you liked this kernel feel free to upvote and leave feedback, thanks! First , let us start with some basic understandings.  BERT  BERT stands for Bidirectional Encoder Representation from Transformers. In a transformer flow if we stack a number of encoders then we get a BERT. It is easier to make BERT understand a language. BERT also has a variety of problems such as Question-Answering , Sentiment Analysis , Text sumamrzation ,etc. Steps to use a BERT model : - Pretraining BERT : To understand language- Fine tune BERT : To help us in our specific task Pretraining BERT - To make BERT learn what is language.- It has two part Masked Langauge Modelling(MLM) and Next Sentence Prediction(NSP).- Both of these problems are trained simultaneously. Fine tuning BERT- It is a quiet fast process.- Only the output parameters are leant from scratch and whereas the rest of the parameters are slightly fine-tuned and not that   much changed which in turn makes the process faster. If you want to read more about BERT. [Click Here.](https://arxiv.org/pdf/1810.04805.pdf) ***  Importing basic libraries<jupyter_code># We will use the official tokenization script created by the Google team
!wget --quiet https://raw.githubusercontent.com/tensorflow/models/master/official/nlp/bert/tokenization.py
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Model
from tensorflow.keras.callbacks import ModelCheckpoint
import tensorflow_hub as hub

import tokenization
train = pd.read_csv("/kaggle/input/nlp-getting-started/train.csv")
test = pd.read_csv("/kaggle/input/nlp-getting-started/test.csv")
submission = pd.read_csv("/kaggle/input/nlp-getting-started/sample_submission.csv")<jupyter_output><empty_output><jupyter_text>Defining helpful functions<jupyter_code>def bert_encode(texts, tokenizer, max_len=512):
    all_tokens = []
    all_masks = []
    all_segments = []
    
    for text in texts:
        text = tokenizer.tokenize(text)
            
        text = text[:max_len-2]
        input_sequence = ["[CLS]"] + text + ["[SEP]"]
        pad_len = max_len - len(input_sequence)
        
        tokens = tokenizer.convert_tokens_to_ids(input_sequence)
        tokens += [0] * pad_len
        pad_masks = [1] * len(input_sequence) + [0] * pad_len
        segment_ids = [0] * max_len
        
        all_tokens.append(tokens)
        all_masks.append(pad_masks)
        all_segments.append(segment_ids)
    
    return np.array(all_tokens), np.array(all_masks), np.array(all_segments)
def build_model(bert_layer, max_len=512):
    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name="input_word_ids")
    input_mask = Input(shape=(max_len,), dtype=tf.int32, name="input_mask")
    segment_ids = Input(shape=(max_len,), dtype=tf.int32, name="segment_ids")

    _, sequence_output = bert_layer([input_word_ids, input_mask, segment_ids])
    clf_output = sequence_output[:, 0, :]
    out = Dense(1, activation='sigmoid')(clf_output)
    
    model = Model(inputs=[input_word_ids, input_mask, segment_ids], outputs=out)
    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])
    
    return model<jupyter_output><empty_output><jupyter_text>Data cleaning<jupyter_code>def lowercase_text(text):
    return text.lower()

train.text=train.text.apply(lambda x: lowercase_text(x))
test.text=test.text.apply(lambda x: lowercase_text(x))
def remove_noise(text):
    text = re.sub('\[.*?\]', '', text)
    text = re.sub('https?://\S+|www\.\S+', '', text)
    text = re.sub('<.*?>+', '', text)
    text = re.sub('[%s]' % re.escape(string.punctuation), '', text)
    text = re.sub('\n', '', text)
    text = re.sub('\w*\d\w*', '', text)
    return text

train.text=train.text.apply(lambda x: remove_noise(x))
test.text=test.text.apply(lambda x: remove_noise(x))
train.text.head(5)<jupyter_output><empty_output><jupyter_text>Pre-training BERT- First we will load bert from tensorhub- From the bert-layer we will load the tokenizer- We will encode and convert the data into Bert-input form<jupyter_code>module_url = "https://tfhub.dev/tensorflow/bert_en_uncased_L-24_H-1024_A-16/1"
bert_layer = hub.KerasLayer(module_url, trainable=True)
vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()
do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
tokenizer = tokenization.FullTokenizer(vocab_file, do_lower_case)
train_input = bert_encode(train.text.values, tokenizer, max_len=160)
test_input = bert_encode(test.text.values, tokenizer, max_len=160)
train_labels = train.target.values<jupyter_output><empty_output><jupyter_text>Modelling<jupyter_code>model = build_model(bert_layer, max_len=160)
model.summary()
checkpoint = ModelCheckpoint('model.h5', monitor='val_loss', save_best_only=True)

train_history = model.fit(
    train_input, train_labels,
    validation_split=0.2,
    epochs=5,
    callbacks=[checkpoint],
    batch_size=15
)
metrics=pd.DataFrame(model.history.history)
metrics
import matplotlib.pyplot as plt
metrics[['loss' , 'val_loss']].plot()
plt.show()
metrics[['accuracy' , 'val_accuracy']].plot()
plt.show()
model.load_weights('model.h5')
test_pred = model.predict(test_input)
submission['target'] = test_pred.round().astype(int)
submission.to_csv('submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11148325358851674, 'fraction_numerical': 0.009888357256778309, 'mean_word_length': 4.833488372093023, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '453114', 'license': 'Apache-2.0', 'n_tokens_mistral': 2081, 'n_tokens_neox': 2015, 'n_words': 547, 'path': '.ipynb_checkpoints/Bert_model-checkpoint.ipynb', 'repo_name': 'friskycodeur/NLP-disaster-tweets', 'size': '6270'}