<jupyter_start><jupyter_text>Data CleaningAs we know,twitter tweets always have to be cleaned before we go onto modelling.So we will do some basic cleaning such as spelling correction,removing punctuations,removing html tags and emojis etc.So let's start.<jupyter_code>tweet, test = dataset_to_df()
df=pd.concat([tweet,test])
df.shape<jupyter_output><empty_output><jupyter_text>Removing urls<jupyter_code>example="New competition launched :https://www.kaggle.com/c/nlp-getting-started"
def remove_URL(text):
    url = re.compile(r'https?://\S+|www\.\S+')
    return url.sub(r'',text)

remove_URL(example)
df['text']=df['text'].apply(lambda x : remove_URL(x))<jupyter_output><empty_output><jupyter_text>Removing HTML tags<jupyter_code>example = """<div>
<h1>Real or Fake</h1>
<p>Kaggle </p>
<a href="https://www.kaggle.com/c/nlp-getting-started">getting started</a>
</div>"""
def remove_html(text):
    html=re.compile(r'<.*?>')
    return html.sub(r'',text)
print(remove_html(example))
df['text']=df['text'].apply(lambda x : remove_html(x))<jupyter_output><empty_output><jupyter_text>Romoving Emojis<jupyter_code># Reference : https://gist.github.com/slowkow/7a7f61f495e3dbb7e3d767f97bd7304b
def remove_emoji(text):
    emoji_pattern = re.compile("["
                           u"\U0001F600-\U0001F64F"  # emoticons
                           u"\U0001F300-\U0001F5FF"  # symbols & pictographs
                           u"\U0001F680-\U0001F6FF"  # transport & map symbols
                           u"\U0001F1E0-\U0001F1FF"  # flags (iOS)
                           u"\U00002702-\U000027B0"
                           u"\U000024C2-\U0001F251"
                           "]+", flags=re.UNICODE)
    return emoji_pattern.sub(r'', text)

remove_emoji("Omg another Earthquake 😔😔")
df['text']=df['text'].apply(lambda x: remove_emoji(x))<jupyter_output><empty_output><jupyter_text>Removing punctuations<jupyter_code>def remove_punct(text):
    table=str.maketrans('','',string.punctuation)
    return text.translate(table)

example="I am a #king"
print(remove_punct(example))
df['text']=df['text'].apply(lambda x : remove_punct(x))<jupyter_output><empty_output><jupyter_text>GloVe for Vectorization Here we will use GloVe pretrained corpus model to represent our words.It is available in 3 varieties :50D ,100D and 200 Dimentional.We will try 100 D here.<jupyter_code>def create_corpus(df):
    corpus=[]
    for tweet in tqdm(df['text']):
        words=[word.lower() for word in word_tokenize(tweet) if((word.isalpha()==1) & (word not in stop))]
        corpus.append(words)
    return corpus
        
        
corpus=create_corpus(df)
embedding_dict={}
with open('../data/raw/glove.6B.100d.txt','r') as f:
    for line in f:
        values=line.split()
        word=values[0]
        vectors=np.asarray(values[1:],'float32')
        embedding_dict[word]=vectors
f.close()
MAX_LEN=50
tokenizer_obj=Tokenizer()
tokenizer_obj.fit_on_texts(corpus)
sequences=tokenizer_obj.texts_to_sequences(corpus)

tweet_pad=pad_sequences(sequences,maxlen=MAX_LEN,truncating='post',padding='post')
word_index=tokenizer_obj.word_index
print('Number of unique words:',len(word_index))
num_words=len(word_index)+1
embedding_matrix=np.zeros((num_words,100))

for word,i in tqdm(word_index.items()):
    if i > num_words:
        continue
    
    emb_vec=embedding_dict.get(word)
    if emb_vec is not None:
        embedding_matrix[i]=emb_vec
            
# pd.to_csv("data/processed/em", header=None, index=None)
tweet_pad[0]<jupyter_output><empty_output><jupyter_text>Baseline Model<jupyter_code>model=Sequential()

embedding=Embedding(num_words,100,embeddings_initializer=Constant(embedding_matrix),
                   input_length=MAX_LEN,trainable=False)

model.add(embedding)
model.add(SpatialDropout1D(0.2))
model.add(LSTM(64, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))


optimzer=Adam(learning_rate=1e-5)

model.compile(loss='binary_crossentropy',optimizer=optimzer,metrics=['accuracy'])


model.summary()
train=tweet_pad[:tweet.shape[0]]
test=tweet_pad[tweet.shape[0]:]
print(test.shape)
X_train,X_test,y_train,y_test=train_test_split(train,tweet['target'].values,test_size=0.15)
print('Shape of train',X_train.shape)
print("Shape of Validation ",X_test.shape)
history=model.fit(X_train,y_train,batch_size=4,epochs=15,validation_data=(X_test,y_test),verbose=2)<jupyter_output>Train on 6471 samples, validate on 1142 samples
Epoch 1/15
 - 52s - loss: 0.6916 - accuracy: 0.5690 - val_loss: 0.6895 - val_accuracy: 0.5779
Epoch 2/15
 - 47s - loss: 0.6867 - accuracy: 0.5690 - val_loss: 0.6759 - val_accuracy: 0.5779
Epoch 3/15
 - 49s - loss: 0.6177 - accuracy: 0.6758 - val_loss: 0.5437 - val_accuracy: 0.7680
Epoch 4/15
 - 47s - loss: 0.5700 - accuracy: 0.7399 - val_loss: 0.5179 - val_accuracy: 0.7732
Epoch 5/15
 - 46s - loss: 0.5608 - accuracy: 0.7458 - val_loss: 0.5107 - val_accuracy: 0.7697
Epoch 6/15
 - 45s - loss: 0.5506 - accuracy: 0.7540 - val_loss: 0.5070 - val_accuracy: 0.7767
Epoch 7/15
 - 44s - loss: 0.5405 - accuracy: 0.7563 - val_loss: 0.5041 - val_accuracy: 0.7767
Epoch 8/15
 - 45s - loss: 0.5424 - accuracy: 0.7558 - val_loss: 0.5033 - val_accuracy: 0.7793
Epoch 9/15
 - 44s - loss: 0.5474 - accuracy: 0.7509 - val_loss: 0.5023 - val_accuracy: 0.7820
Epoch 10/15
 - 45s - loss: 0.5371 - accuracy: 0.7609 - val_loss: 0.5007 - val_accuracy: 0.7811
Epoch 11/15[...]<jupyter_text>Save the model and tokenizer<jupyter_code>model.save("../models/model.h5")
with open('../models/tokenizer.pickle', 'wb') as handle:
    pickle.dump(tokenizer_obj, handle, protocol=pickle.HIGHEST_PROTOCOL)<jupyter_output><empty_output><jupyter_text>Making our submission<jupyter_code># sample_sub=pd.read_csv('../data/raw/sample_submission.csv')
# y_pre=model.predict(test)
# y_pre=np.round(y_pre).astype(int).reshape(3263)
# sub=pd.DataFrame({'id':sample_sub['id'].values.tolist(),'target':y_pre})
# sub.to_csv('../data/processed/submission.csv',index=False)

#sub.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14812967581047382, 'fraction_numerical': 0.06949293433083957, 'mean_word_length': 4.474067333939946, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 46, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '182492', 'license': 'MIT', 'n_tokens_mistral': 2584, 'n_tokens_neox': 2300, 'n_words': 375, 'path': 'notebooks/prepare_train.nbconvert.ipynb', 'repo_name': 'itsderek23/nlp_disaster_tweets', 'size': '6015'}