<jupyter_start><jupyter_text>Define the problem  How can we identify the disasters through tweets? Twitter has become an important communication channel in times of emergency. The ubiquitousness of smartphones enables people to announce an emergency they’re observing in real-time. Because of this, more agencies are interested in programatically monitoring Twitter (i.e. disaster relief organizations and news agencies)But, it’s not always clear whether a person’s words are actually announcing a disaster. Through this project we want to solve this problem using some advanced technologies of NLP and accuratley classify between emergency and non-emergency tweets.<jupyter_code>#Import libraries
import numpy as np
import pandas as pd
import os
import io
import json

import sys
sys.path.insert(1, '../src')

from features import build_features as bf
from visualization import visualize as vs
from models import train_model as tm

from os import path
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator

#Import visualization libraries
import matplotlib.pyplot as plt
import seaborn as sns
from collections import defaultdict
import string

import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

#Tensorflow libraries
from tensorflow.keras import Sequential
from tensorflow.keras.layers import LSTM, Dense, Bidirectional,GlobalMaxPooling1D, Embedding,BatchNormalization, GlobalAveragePooling1D, Dropout
from tensorflow.keras.models import model_from_json<jupyter_output><empty_output><jupyter_text>Loading the Data<jupyter_code>tweet_data = pd.read_csv('../data/raw/train.csv')
tweet_data.head()<jupyter_output><empty_output><jupyter_text>Checking out null values Missing values<jupyter_code>tweet_data.isnull().sum()<jupyter_output><empty_output><jupyter_text>Missing values Percentage<jupyter_code>(tweet_data.isnull().sum()/len(tweet_data))*100<jupyter_output><empty_output><jupyter_text>Cleaning the data The most values we are missing is location and we dont have any other data to replace it.Lets replace the missing key words with "NAN" and drop the location column<jupyter_code>def cleaning_data(tweet_data):
    tweet_data = tweet_data.drop(labels=["location", "id"], axis=1)
    tweet_data['keyword'] = tweet_data['keyword'].replace(np.nan, "NAN")
    print((tweet_data.isnull().sum()/len(tweet_data))*100)
    return tweet_data
tweet_data = cleaning_data(tweet_data)<jupyter_output>keyword    0.0
text       0.0
target     0.0
dtype: float64<jupyter_text>Exploring the Data (EDA) Lets check out if we equal proportion of the data<jupyter_code>tweet_data.groupby("target").count()<jupyter_output><empty_output><jupyter_text>It looks like we have slightly more non-disaster value compared to disaster value butthere is no wide gap.<jupyter_code>vs.proportion(tweet_data)<jupyter_output><empty_output><jupyter_text>We can see that overall not disaster tweets have more wordscompared to disaster tweets.<jupyter_code>vs.word_count(tweet_data)<jupyter_output><empty_output><jupyter_text>The avg word length is very similar.<jupyter_code>vs.avg_word_length(tweet_data)<jupyter_output><empty_output><jupyter_text>We can see that disaster tweets usally have "Fire, Suicide bombing, car crash e.t.c" and non disaster has more words like "love, people,good e.t.c "         We can see the keywords are a better indicator distinguishing between disaster and non disaster.<jupyter_code>vs.word_cloud(tweet_data)
def create_corpus(target):
    corpus = []
    for words in tweet_data[tweet_data['target'] == target]['text'].str.split():
        for word in words:
            corpus.append(word)
    return corpus
corpus = create_corpus(0)
vs.punctuations(corpus)
corpus = create_corpus(1)
vs.punctuations(corpus, color='red')<jupyter_output><empty_output><jupyter_text>--Preprocessing the data--  Preprocessing the data is a very important step in NLP  Step 1: Shuffling the Data It is very important to shuffle the data to make sure that when we split the data we do not end up with imblanced classes that is we dont want to many emergency examples in our training or test data vice versa. Step 2: Combine the keywords with text After closley looking at the data we realized that 'keyword' column is actually a subspace of the 'text'. For example, keyword: bomb text: There is a man with a bomb and the bomb is placed in the building  As you can see the bomb is represented in the text multiple times as well as the keyword.     Step 3: Removing Stop words, Links and emojisStop words are the words we use quite often in our language but hold a little are almost no meaning.     For example, 'The', 'by', 'a', e.t.c  compared to  'person', 'bomd', 'love', e.t.c  By getting rid of this stop words we save a lot of computational time as well as help our NLP model to easily understand https links and emojis are too complex for our models to make sense.   Step 4: Splitting the Data     We split our train data into train and validation with a 80/20 split as the test data is provided to test it on Kaggle we do not need it.    Step 5: Tokenization          Tokenization plays a key role in NLP algorithms. We are going to use the tensorflow inbuilt tokenizer library.            What a tokenizer does is it maps our words in texts to numbers            For example, Text: "I love you"            will be translated to {"I":1, "love":2, "you":3, "OOV":4}            We are going to use only the texts form our training as we do not want any data lekage in our solution.         Step 6: Sequencing         We are going to turn our words into sequences "I love you" will be translated to [1,2,3] Step 7: Padding             As we want all our data to have the same dimensions we are going to pad the smaller sentenses with 0 at the end.<jupyter_code>def pre_processing_data(tweet_data):
    '''
    Tokenizes our data.
    '''
    #random shuffle the data
    tweet_data = tweet_data.sample(frac=1)
    
    #Merging the text with key word
    tweet_data_text = tweet_data['keyword'].str.cat(tweet_data['text'], sep=' ')
    
    #Clean the data
    tweet_data_text = tweet_data_text.apply(lambda text: bf.clean_text(text))
    
    #Remove emojis
    tweet_data_text = tweet_data_text.apply(lambda text: bf.remove_emoji(text))
    
    #Removing Stop words
    tweet_data_text = bf.remove_stopwords(tweet_data_text)
    
    #Train val split
    (train_text, train_labels), (val_text, val_labels) = bf.train_val_split(tweet_data_text, tweet_data['target'])
    
    max_len = len(max(train_text, key= lambda sentence: len(sentence.split())))
    
    #Tokenization
    text_tokenizer = Tokenizer(oov_token='<OOV>')
    
    text_tokenizer.fit_on_texts(train_text)
    
    print(len(text_tokenizer.word_index))
    #Sequencing
    sequences_text_train = text_tokenizer.texts_to_sequences(train_text)
    sequences_labels_train = np.array(train_labels)
    
    sequences_text_val = text_tokenizer.texts_to_sequences(val_text)
    sequences_labels_val = np.array(val_labels)

    #Padding
    
    padded_text_train = pad_sequences(sequences_text_train, padding='post', maxlen=max_len)
    padded_text_val = pad_sequences(sequences_text_val, padding='post', maxlen=max_len)
    
    #Saving the tokenizer
    bf.save_tokenizer(text_tokenizer)
    
    return (padded_text_train, sequences_labels_train),(padded_text_val, sequences_labels_val)
(train_texts, train_labels), (val_texts, val_labels) = pre_processing_data(tweet_data)<jupyter_output>14518<jupyter_text>--Establishing a Baseline--<jupyter_code>def danger_words():
    '''
    returns The first 5 keywords that are closley associated
    with emergency
    '''
    danger_keyword = tweet_data[tweet_data['target'] == 1].groupby('keyword').count()
    danger_keyword = danger_keyword.sort_values('text',ascending=False).iloc[1:6]
    danger_keyword = danger_keyword.index.values
    return danger_keyword
def baseline(test_data, danger_keyword):
    for word in danger_keyword:
        if(test_data['keyword'] == word):
            test_data['target'] = 1
    return test_data<jupyter_output><empty_output><jupyter_text>-- Evaluating the baseline--<jupyter_code>def eval_base():
    '''
    Evaluating the baseline model.
    '''
    danger_keyword = danger_words()
    test_data = pd.read_csv('../data/raw/test.csv')
    test_data['keyword'] = test_data['keyword'].replace(np.nan, "NAN")
    test_data['target'] = 0
    test_data = test_data.apply(lambda x: baseline(x, danger_keyword),axis=1)
    test_data = test_data.drop(['location', 'keyword', 'text'], axis=1)
    test_data.set_index('id').to_csv('../data/external/submission_base.csv')
eval_base()<jupyter_output><empty_output><jupyter_text>My First submission with the baseline model resulted in a 0.588 mean F-1 score.    Lets try to improve this with squence models. and was placed at 1131 of 1273 teams  --Developing models-- 1. LSTM Model  Building the model What is an LSTM Model? LSTM is an abbrevation for Long Short Term Memory(LSTM) what this means is that in our neural networks we have a memory cell that can store words that might play a key factor in determining what the output might be at the end of the sentence.  What is an Word Embedding? A word embedding is a class of approaches for representing words and documents using a dense vector representation.It is an improvement over more the traditional bag-of-word model encoding schemes where large sparse vectors were used to represent each word or to score each word within a vector to represent an entire vocabulary. These representations were sparse because the vocabularies were vast and a given word or document would be represented by a large vector comprised mostly of zero values What is batchnormalization?Batch normalization (also known as batch norm) is a method used to make artificial neural networks faster and more stable through normalization of the input layer by re-centering and re-scaling Callbacks We used early stopping callbacks with a patience level of 10 so we do not overfit our validation data<jupyter_code>def lstm_model(input_length, dim=[128,64,32], train=(), val=()):
    '''
    LSTM Model
    '''
    with open('../src/models/tokenizer.json') as f:
        data = json.load(f)
        tokenizer = tf.keras.preprocessing.text.tokenizer_from_json(data)
    
    vocab_len = len(tokenizer.word_index)+1
    
    for out in dim:
        #model
        model = Sequential()
        #Embedding
        model.add(Embedding(vocab_len, out, input_length=input_length))
        #Layer 1
        model.add(Bidirectional(LSTM(input_length, return_sequences=True)))
        model.add(GlobalAveragePooling1D())
        model.add(BatchNormalization())
        model.add(Dropout(0.5))
        
        #Final layer
        model.add(Dense(1, activation='sigmoid'))

        #Early stopping call backs
        callback = tm.callbacks()

        #Complile
        model.compile(loss='binary_crossentropy', optimizer='adam', 
                              metrics=['accuracy'])

        #fit
        history = model.fit(train[0], train[1], validation_data=(val[0], val[1]), epochs =100,
                         callbacks=[callback])

        #Visualize
        vs.training_visualize(history, title='model_LSTM_{}'.format(out))

        #Save model
        tm.save_model(model, title='model_LSTM_{}'.format(out))
        
lstm_model(input_length = train_texts.shape[1], train=(train_texts, train_labels),
          val=(val_texts, val_labels))<jupyter_output><empty_output><jupyter_text>The LSTM submission resulted in a F1-Score of 0.76892 and was placed as 847 of 1273 teams which is an improvment from baseline model but let's see if we can do better. ![0](../reports/performance/model_LSTM_128.png) ![0](../reports/performance/model_LSTM_64.png) ![0](../reports/performance/model_LSTM_32.png)<jupyter_code>#2. Glove and LSTM<jupyter_output><empty_output><jupyter_text>What is Glove?GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.<jupyter_code>def glove_LSTM(input_len, train=(), val=()):
    '''
    Glove LSTM
    '''
    with open('../src/models/tokenizer.json') as f:
        data = json.load(f)
        tokenizer = tf.keras.preprocessing.text.tokenizer_from_json(data)
        
    vocab_len = len(tokenizer.word_index)+1
    
    dimensions = [100,200,300]
    for dim in dimensions:
        embedding_dict = dict()
        glove_file = open('../data/interim/glove/glove.6B.{}d.txt'.format(dim), encoding='utf8')
        
        for line in glove_file:
            records = line.split()
            word = records[0]
            vector_dimensions = np.asarray(records[1:], dtype='float32')
            embedding_dict[word] = vector_dimensions   
        glove_file.close()
        
        embedding_matrix = np.zeros((vocab_len,dim))
        for word, index in tokenizer.word_index.items():
            embedding_vector = embedding_dict.get(word)
            if embedding_vector is not None:
                embedding_matrix[index] = embedding_vector
        model = Sequential()
        model.add(Embedding(input_dim=embedding_matrix.shape[0],
                           output_dim = embedding_matrix.shape[1],
                           weights = [embedding_matrix],
                           input_length =input_len ))
        model.add(Bidirectional(LSTM(input_len, 
                                     return_sequences = True,
                                    recurrent_dropout=0.2)))
        model.add(GlobalAveragePooling1D())
        model.add(BatchNormalization())
        model.add(Dropout(0.5))
        
        model.add(Dense(128, activation='relu'))
        model.add(Dropout(0.4))
        
        model.add(Dense(64, activation='relu'))
        model.add(Dropout(0.3))
        
        model.add(Dense(1, activation='sigmoid'))
        
        model.compile(optimizer='adam', loss='binary_crossentropy', 
                      metrics=['accuracy'])
        #Early stopping call backs
        callback = tm.callbacks()
        
        #fit
        history = model.fit(train[0], train[1], validation_data=(val[0], val[1]), epochs =100,
                         callbacks=[callback])

        #Visualize
        vs.training_visualize(history, title='glove_LSTM_{}'.format(dim))

        #Save model
        tm.save_model(model, title='glove_LSTM_{}'.format(dim))
glove_LSTM(input_len = train_texts.shape[1], train=(train_texts, train_labels),
          val=(val_texts, val_labels))<jupyter_output><empty_output><jupyter_text>![0](../reports/performance/glove_LSTM_100.png) ![0](../reports/performance/glove_LSTM_200.png) ![0](../reports/performance/glove_LSTM_300.png)  The glove LSTM submission resulted in a F1-Score of 0.80171 and was places 601 of 1273 which is an improvment from LSTM model but let's see if we can do better.<jupyter_code>#3. Bert<jupyter_output><empty_output><jupyter_text>To run bert we need high computational power and access to large space of memory. So instead we decided to run it on Kaggle platform and submit the result from there  We were able to achieve an 0.8271 F1-Score were placed in 336 of 1268 teams in the competition   --Evaluating the models--<jupyter_code>def eval_tokenizer():
    #Replace NAN's and concat columns
    test_data = pd.read_csv('../data/raw/test.csv')
    test_data['keyword'] = test_data['keyword'].replace(np.nan, "NAN")
    test_text = test_data['keyword'].str.cat(test_data['text'], sep=' ')
    
    #Clean the data
    test_text = test_text.apply(lambda text: bf.clean_text(text))
    
    #Remove emojis
    test_text = test_text.apply(lambda text: bf.remove_emoji(text))
    
    #Removing Stop words
    test_text = bf.remove_stopwords(test_text)
    # Clean the data
    
    #Tokenize
    with open('../src/models/tokenizer.json') as f:
        data = json.load(f)
        tokenizer = tf.keras.preprocessing.text.tokenizer_from_json(data)
    
    sequence_text = tokenizer.texts_to_sequences(test_text)
    padded_text = pad_sequences(sequence_text, padding='post', maxlen=train_texts.shape[1])
    return padded_text, test_data
def lstm_eval():
    text_seq, test_data = eval_tokenizer()
    dim = [128, 64, 32]
    for out in dim:
        json_file = open('../src/models/saved_models/model_model_LSTM_{}.json'.format(out),'r')
        loaded_model_json = json_file.read()
        json_file.close()
        loaded_model = model_from_json(loaded_model_json)
        loaded_model.load_weights('../src/models/saved_models/model_model_LSTM_{}.h5'.format(out))
        print("Loaded model from disk")
        loaded_model.compile(loss='binary_crossentropy',optimizer='adam', metrics=['accuracy'])
        predictions = loaded_model.predict_classes(text_seq)
        pred = pd.DataFrame(predictions, columns=['target'])
        pred = pd.concat([pred, test_data['id']], axis=1)
        pred.set_index('id').to_csv('../data/external/submission_LSTM_{}.csv'.format(out))
lstm_eval()
def glove_lstm_eval():
    text_seq, test_data = eval_tokenizer()
    dimensions = [100,200,300]
    for out in dimensions:
        json_file = open('../src/models/saved_models/model_glove_LSTM_{}.json'.format(out),'r')
        loaded_model_json = json_file.read()
        json_file.close()
        loaded_model = model_from_json(loaded_model_json)
        loaded_model.load_weights('../src/models/saved_models/model_glove_LSTM_{}.h5'.format(out))
        print("Loaded model from disk")
        loaded_model.compile(loss='binary_crossentropy',optimizer='adam', metrics=['accuracy'])
        predictions = loaded_model.predict_classes(text_seq)
        pred = pd.DataFrame(predictions, columns=['target'])
        pred = pd.concat([pred, test_data['id']], axis=1)
        pred.set_index('id').to_csv('../data/external/submission_glove_LSTM_{}.csv'.format(out))
        
glove_lstm_eval()<jupyter_output>Loaded model from disk
WARNING:tensorflow:`model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype("int32")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).
Loaded model from disk
WARNING:tensorflow:`model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype("int32")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).
Loaded model from disk
WARNING:tensorflow:`model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.a[...]
{'fraction_non_alphanumeric': 0.09780449555671719, 'fraction_numerical': 0.013120752744380555, 'mean_word_length': 3.8432911392405065, 'pattern_counts': {'":': 4, '<': 71, '<?xml version=': 0, '>': 73, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '545013', 'license': 'MIT', 'n_tokens_mistral': 5977, 'n_tokens_neox': 5711, 'n_words': 1663, 'path': 'notebooks/Untitled.ipynb', 'repo_name': 'sa616473/real_or_not', 'size': '19130'}