<jupyter_start><jupyter_text>Import necessary packages<jupyter_code>import json
import tensorflow as tf

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences<jupyter_output><empty_output><jupyter_text>Set the hyper-parametres<jupyter_code>vocab_size = 10000
embedding_dim = 16
max_length = 100
trunc_type='post'
padding_type='post'
oov_tok = "<OOV>"
training_size = 20000<jupyter_output><empty_output><jupyter_text>Download the dataset<jupyter_code>!wget --no-check-certificate \
    https://storage.googleapis.com/laurencemoroney-blog.appspot.com/sarcasm.json \
    -O /tmp/sarcasm.json<jupyter_output>--2019-09-27 17:18:03--  https://storage.googleapis.com/laurencemoroney-blog.appspot.com/sarcasm.json
Resolving storage.googleapis.com (storage.googleapis.com)... 192.168.3.11, fd00:c2b6:b24b:be67:2827:688d:e6a1:6a3b
Connecting to storage.googleapis.com (storage.googleapis.com)|192.168.3.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5643545 (5.4M) [application/json]
Saving to: ‘/tmp/sarcasm.json’


/tmp/sarcasm.json     0%[                    ]       0  --.-KB/s               
/tmp/sarcasm.json   100%[===================>]   5.38M  --.-KB/s    in 0.07s   

2019-09-27 17:18:03 (72.7 MB/s) - ‘/tmp/sarcasm.json’ saved [5643545/5643545]<jupyter_text>Open the json file and extract sentences and labels in the form of lists<jupyter_code>with open("/tmp/sarcasm.json", 'r') as f:
    datastore = json.load(f)

sentences = []
labels = []

for item in datastore:
    sentences.append(item['headline'])
    labels.append(item['is_sarcastic'])
training_sentences = sentences[0:training_size]
testing_sentences = sentences[training_size:]
training_labels = labels[0:training_size]
testing_labels = labels[training_size:]<jupyter_output><empty_output><jupyter_text>Do the tokenization and pad the sequences<jupyter_code>tokenizer = Tokenizer(num_words=vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(training_sentences)

word_index = tokenizer.word_index

training_sequences = tokenizer.texts_to_sequences(training_sentences)
training_padded = pad_sequences(training_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)

testing_sequences = tokenizer.texts_to_sequences(testing_sentences)
testing_padded = pad_sequences(testing_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)<jupyter_output><empty_output><jupyter_text>Our Keras Model<jupyter_code>model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
    tf.keras.layers.GlobalAveragePooling1D(),
    tf.keras.layers.Dense(24, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Print the summary of our model<jupyter_code>model.summary()<jupyter_output>Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, 100, 16)           160000    
_________________________________________________________________
global_average_pooling1d_1 ( (None, 16)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 24)                408       
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 25        
=================================================================
Total params: 160,433
Trainable params: 160,433
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Set the number of epochs and fit the model<jupyter_code>num_epochs = 30
history = model.fit(training_padded, training_labels, epochs=num_epochs, validation_data=(testing_padded, testing_labels), verbose=2)<jupyter_output>Train on 20000 samples, validate on 6709 samples
Epoch 1/30
20000/20000 - 3s - loss: 0.6703 - acc: 0.5712 - val_loss: 0.5937 - val_acc: 0.6667
Epoch 2/30
20000/20000 - 3s - loss: 0.4293 - acc: 0.8278 - val_loss: 0.3795 - val_acc: 0.8435
Epoch 3/30
20000/20000 - 3s - loss: 0.3033 - acc: 0.8789 - val_loss: 0.3549 - val_acc: 0.8483
Epoch 4/30
20000/20000 - 3s - loss: 0.2514 - acc: 0.9033 - val_loss: 0.3436 - val_acc: 0.8565
Epoch 5/30
20000/20000 - 2s - loss: 0.2147 - acc: 0.9172 - val_loss: 0.3476 - val_acc: 0.8557
Epoch 6/30
20000/20000 - 2s - loss: 0.1879 - acc: 0.9280 - val_loss: 0.3610 - val_acc: 0.8526
Epoch 7/30
20000/20000 - 2s - loss: 0.1681 - acc: 0.9363 - val_loss: 0.3727 - val_acc: 0.8550
Epoch 8/30
20000/20000 - 2s - loss: 0.1469 - acc: 0.9474 - val_loss: 0.4311 - val_acc: 0.8375
Epoch 9/30
20000/20000 - 2s - loss: 0.1330 - acc: 0.9521 - val_loss: 0.4159 - val_acc: 0.8501
Epoch 10/30
20000/20000 - 2s - loss: 0.1191 - acc: 0.9582 - val_loss: 0.4539 - val_acc: 0.8438
Epoch 11/3[...]<jupyter_text>Plot the loss using matplotlib<jupyter_code>import matplotlib.pyplot as plt

def plot_graphs(history, string):
  plt.plot(history.history[string])
  plt.plot(history.history['val_'+string])
  plt.xlabel("Epochs")
  plt.ylabel(string)
  plt.legend([string, 'val_'+string])
  plt.show()

plot_graphs(history, "loss")<jupyter_output><empty_output><jupyter_text>Decoding our training sentences after tokenizing and padding them<jupyter_code>reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])

def decode_sentence(text):
    return ' '.join([reverse_word_index.get(i, '?') for i in text])

print(decode_sentence(training_padded[0]))
print(training_sentences[2])
print(labels[2])<jupyter_output>former <OOV> store clerk sues over secret 'black <OOV> for minority shoppers ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
mom starting to fear son's web series closest thing she will have to grandchild
1<jupyter_text>Get the number of words and also the dimensions of our embeddings<jupyter_code>e = model.layers[0]
weights = e.get_weights()[0]
print(weights.shape) # shape: (vocab_size, embedding_dim)<jupyter_output>(10000, 16)<jupyter_text>For Visualization on Tensorboard<jupyter_code>import io

out_v = io.open('vecs.tsv', 'w', encoding='utf-8')
out_m = io.open('meta.tsv', 'w', encoding='utf-8')
for word_num in range(1, vocab_size):
  word = reverse_word_index[word_num]
  embeddings = weights[word_num]
  out_m.write(word + "\n")
  out_v.write('\t'.join([str(x) for x in embeddings]) + "\n")
out_v.close()
out_m.close()
try:
  from google.colab import files
except ImportError:
  pass
else:
  files.download('vecs.tsv')
  files.download('meta.tsv')<jupyter_output><empty_output><jupyter_text>Prediction<jupyter_code>sentence = ["granny starting to fear spiders in the garden might be real", "game of thrones season finale showing this sunday night"]
sequences = tokenizer.texts_to_sequences(sentence)
padded = pad_sequences(sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)
print(model.predict(padded))<jupyter_output>[[0.49796104]
 [0.49761578]]
{'fraction_non_alphanumeric': 0.14714305195690713, 'fraction_numerical': 0.07636710759579982, 'mean_word_length': 4.857827476038339, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '224173', 'license': 'Apache-2.0', 'n_tokens_mistral': 2978, 'n_tokens_neox': 2569, 'n_words': 497, 'path': 'model_on_sarcasm.ipynb', 'repo_name': 'umaralik/NLP', 'size': '7322'}