<jupyter_start><jupyter_text>This notebook will be purely focusing on the modeling part. I will start off by installing the GPU variant of TensorFlow 2.0 (Colab gives you T4 Tesla GPUs for free). **Note** that this notebook does not explore the dataset. If you are interested to know about that check this [EDA notebook](https://github.com/sayakpaul/TF-2.0-Hacks/blob/master/Predicting%20publisher's%20name%20from%20an%20article%20with%20TF%202.0/Modeling_with_TensorFlow_2_0_and_Keras.ipynb).<jupyter_code>!pip install tensorflow-gpu==2.0.0-alpha0<jupyter_output><empty_output><jupyter_text>I have the data serialized already. So, I would just load them and remove the index column which is a result of my inattentiveness.<jupyter_code>import pandas as pd
import numpy as np

train = pd.read_csv('data/train.csv')
valid = pd.read_csv('data/valid.csv')
test = pd.read_csv('data/test.csv')
train.drop('Unnamed: 0', axis='columns', inplace=True)
valid.drop('Unnamed: 0', axis='columns', inplace=True)
test.drop('Unnamed: 0', axis='columns', inplace=True)
train.columns
train.source.value_counts()<jupyter_output><empty_output><jupyter_text>I like to define the constants and their values which would be used throughout the modeling process at the very beginning. Here, I will define:- A mapping for encoding the sources to integers (computers understand numbers)- Total size of the vocabulary that would be formed from the titles- The maximum sequence length which would be needed for the preprocessing steps<jupyter_code># Label encode
CLASSES = {'blogspot': 0, 'github': 1, 'techcrunch': 2, 'nytimes': 3} 
# Maximum vocabulary size used for tokenization
TOP_K = 20000 
# Sentences will be truncated/padded to this length
MAX_SEQUENCE_LENGTH = 50<jupyter_output><empty_output><jupyter_text>Next I would define a small helper function which would take a DataFrame and will - prepare a list of titles from the DstaFrame- take the sources from the DataFrame, map them to integers and append to a NumPy array<jupyter_code>def return_data(df):
  return list(df['title']), np.array(df['source'].map(CLASSES))
# Apply it to the three splits
train_text, train_labels = return_data(train)
valid_text, valid_labels = return_data(valid)
test_text, test_labels = return_data(test)
train_text[0], train_labels[0]
# TensorFlow imports
import tensorflow as tf
from tensorflow.keras.preprocessing import sequence, text
from tensorflow.keras import models
from tensorflow.keras.layers import Dense, Dropout, Embedding, Conv1D, MaxPooling1D, GlobalAveragePooling1D
# Create a vocabulary from training corpus
tokenizer = text.Tokenizer(num_words=TOP_K)
tokenizer.fit_on_texts(train_text)
# Save token dictionary to use during inference
import pickle 

pickle.dump(tokenizer, open('tokenizer.pickled', 'wb'))<jupyter_output><empty_output><jupyter_text>I am going to the `GloVe` embeddings to represent the words in the titles to a dense representation. The embeddings are of more than 650 MB and the GCP team has it stored in a Google Storage Bucket. This is going to be incredibly helpful since it will allow me to directly use it at a very fast speed.<jupyter_code>!gsutil cp gs://cloud-training-demos/courses/machine_learning/deepdive/09_sequence/text_classification/glove.6B.200d.txt glove.6B.200d.txt<jupyter_output>Copying gs://cloud-training-demos/courses/machine_learning/deepdive/09_sequence/text_classification/glove.6B.200d.txt...
| [1 files][661.3 MiB/661.3 MiB]                                                
Operation completed over 1 objects/661.3 MiB.<jupyter_text>I will now define a helper function which will map the words in the titles with respect to the Glove embeddings. In literature this is often referred to as _embedding matrix_.<jupyter_code>def get_embedding_matrix(word_index, embedding_path, embedding_dim):
  embedding_matrix_all = {}
  with open(embedding_path) as f:
    for line in f:  # Every line contains word followed by the vector value
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        embedding_matrix_all[word] = coefs

  # Prepare embedding matrix with just the words in our word_index dictionary
  num_words = min(len(word_index) + 1, TOP_K)
  embedding_matrix = np.zeros((num_words, embedding_dim))

  for word, i in word_index.items():
    if i >= TOP_K:
        continue
    embedding_vector = embedding_matrix_all.get(word)
    if embedding_vector is not None:
        # Words not found in embedding index will be all-zeros.
        embedding_matrix[i] = embedding_vector
  
  return embedding_matrix<jupyter_output><empty_output><jupyter_text>Let's now define the hyperparameters for the network and also define the constants.<jupyter_code>filters=64
dropout_rate=0.2
embedding_dim=200
kernel_size=3
pool_size=3
word_index=tokenizer.word_index
embedding_path = '/content/glove.6B.200d.txt'
embedding_dim=200<jupyter_output><empty_output><jupyter_text>Model buildingHere I am using a CNN which would basically start by convolving on the embeddings fed to it. Locality is important in sequential data and CNNs would allow me to capture that effectively. The trick is to do all the fundamental CNN operations (convolution, pooling) in 1D.<jupyter_code># Create model instance
model = models.Sequential()
num_features = min(len(word_index) + 1, TOP_K)

# Add embedding layer - GloVe embeddings
model.add(Embedding(input_dim=num_features,
              output_dim=embedding_dim,
              input_length=MAX_SEQUENCE_LENGTH,
              weights=[get_embedding_matrix(word_index, 
                              embedding_path, embedding_dim)],
              trainable=True))

model.add(Dropout(rate=dropout_rate))
model.add(Conv1D(filters=filters,
              kernel_size=kernel_size,
              activation='relu',
              bias_initializer='he_normal',
              padding='same'))

model.add(MaxPooling1D(pool_size=pool_size))
model.add(Conv1D(filters=filters * 2,
              kernel_size=kernel_size,
              activation='relu',
              bias_initializer='he_normal',
              padding='same'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(rate=dropout_rate))
model.add(Dense(len(CLASSES), activation='softmax'))

# Compile model with learning parameters.
optimizer = tf.keras.optimizers.Adam(lr=0.001)
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['acc'])
tf.keras.utils.plot_model(model, to_file='cnn_txt_cls.png')<jupyter_output><empty_output><jupyter_text>Data preprocessing: Tokenize + Pad<jupyter_code># Preprocess the train, validation and test sets
# Tokenize and pad sentences
tokenizer = pickle.load( open( "tokenizer.pickled", "rb" ) )

preproc_train = tokenizer.texts_to_sequences(train_text)
preproc_train = sequence.pad_sequences(preproc_train, maxlen=MAX_SEQUENCE_LENGTH)

preproc_valid = tokenizer.texts_to_sequences(valid_text)
preproc_valid = sequence.pad_sequences(preproc_valid, maxlen=MAX_SEQUENCE_LENGTH)

preproc_test = tokenizer.texts_to_sequences(test_text)
preproc_test = sequence.pad_sequences(preproc_test, maxlen=MAX_SEQUENCE_LENGTH)<jupyter_output><empty_output><jupyter_text>Model training<jupyter_code>H = model.fit(preproc_train,
         train_labels,
         validation_data=(preproc_valid, valid_labels),
         batch_size=128,
         epochs=10,
         verbose=1)<jupyter_output>Train on 110070 samples, validate on 13759 samples
Epoch 1/10
110070/110070 [==============================] - 11s 104us/sample - loss: 0.8801 - acc: 0.6222 - val_loss: 0.7268 - val_acc: 0.7067
Epoch 2/10
110070/110070 [==============================] - 11s 99us/sample - loss: 0.6289 - acc: 0.7515 - val_loss: 0.7007 - val_acc: 0.7192
Epoch 3/10
110070/110070 [==============================] - 11s 99us/sample - loss: 0.5289 - acc: 0.7946 - val_loss: 0.7149 - val_acc: 0.7204
Epoch 4/10
110070/110070 [==============================] - 11s 98us/sample - loss: 0.4406 - acc: 0.8307 - val_loss: 0.7593 - val_acc: 0.7166
Epoch 5/10
110070/110070 [==============================] - 11s 96us/sample - loss: 0.3570 - acc: 0.8649 - val_loss: 0.8373 - val_acc: 0.7139
Epoch 6/10
110070/110070 [==============================] - 11s 96us/sample - loss: 0.2857 - acc: 0.8922 - val_loss: 0.9513 - val_acc: 0.7107
Epoch 7/10
110070/110070 [==============================] - 11s 96us/sample - loss: 0.2350 - acc[...]<jupyter_text>The model overfits :( But still let's evaluate it.<jupyter_code>model.evaluate(preproc_test, test_labels)<jupyter_output>13759/13759 [==============================] - 1s 98us/sample - loss: 1.5162 - acc: 0.6936<jupyter_text>I am now interested in predicting on a few latest title samples which I gathered from Hacker News.<jupyter_code># Helper function to test on single samples
def test_on_single_sample(text):
  category = None
  text_tokenized = tokenizer.texts_to_sequences(text)
  text_tokenized = sequence.pad_sequences(text_tokenized,maxlen=50)
  prediction = int(model.predict_classes(text_tokenized))
  for key, value in CLASSES.items():
    if value==prediction:
      category=key
  
  return category
# Prepare the samples
github=['Invaders game in 512 bytes']
nytimes = ['<NAME> Promises $500M to Help End Coal']
techcrunch = ['Facebook plans June 18th cryptocurrency debut']
blogspot = ['Android Security: A walk-through of SELinux']
for sample in [github, nytimes, techcrunch, blogspot]:
  print(test_on_single_sample(sample))<jupyter_output>github
techcrunch
techcrunch
blogspot
{'fraction_non_alphanumeric': 0.11701009251471825, 'fraction_numerical': 0.0443650126156434, 'mean_word_length': 4.941911305434104, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '147403', 'license': 'Apache-2.0', 'n_tokens_mistral': 3328, 'n_tokens_neox': 2978, 'n_words': 824, 'path': "Predicting publisher's name from an article with TF 2.0/Modeling_with_TensorFlow_2_0_and_Keras.ipynb", 'repo_name': 'joe733/TF-2.0-Hacks', 'size': '9523'}