title using pretrained word embeddings fchollethttps twitter comfchollet date created 20200505 last modified 20200505 description text classification on the newsgroup20 dataset using pretrained glove word embeddings accelerator gpu setup only the tensorflow backend supports string inputs introduction in this example we show how to train a text classification model that uses pretrained word embeddings we ll work with the newsgroup20 dataset a set of 20 000 message board messages belonging to 20 different topic categories for the pretrained word embeddings we ll use glove embeddingshttp nlp stanford eduprojectsglove download the newsgroup20 data let s take a look at the data here s a example of what one file contains as you can see there are header lines that are leaking the file s category either explicitly the first line is literally the category name or implicitly e g via the organization filed let s get rid of the headers there s actually one category that doesn t have the expected number of files but the difference is small enough that the problem remains a balanced classification problem shuffle and split the data into training validation sets shuffle the data extract a training validation split create a vocabulary index let s use the textvectorization to index the vocabulary found in the dataset later we ll use the same layer instance to vectorize the samples our layer will only consider the top 20 000 words and will truncate or pad sequences to be actually 200 tokens long you can retrieve the computed vocabulary used via vectorizer getvocabulary let s print the top 5 words let s vectorize a test sentence as you can see the gets represented as 2 why not 0 given that the was the first word in the vocabulary that s because index 0 is reserved for padding and index 1 is reserved for out of vocabulary tokens here s a dict mapping words to their indices as you can see we obtain the same encoding as above for our test sentence load pretrained word embeddings let s download pretrained glove embeddings a 822m zip file you ll need to run the following commands shell wget https downloads cs stanford edunlpdataglove 6b zip unzip q glove 6b zip the archive contains textencoded vectors of various sizes 50dimensional 100dimensional 200dimensional 300dimensional we ll use the 100d ones let s make a dict mapping words strings to their numpy vector representation now let s prepare a corresponding embedding matrix that we can use in a keras embedding layer it s a simple numpy matrix where entry at index i is the pretrained vector for the word of index i in our vectorizer s vocabulary prepare embedding matrix words not found in embedding index will be allzeros this includes the representation for padding and oov next we load the pretrained word embeddings matrix into an embedding layer note that we set trainablefalse so as to keep the embeddings fixed we don t want to update them during training build the model a simple 1d convnet with global max pooling and a classifier at the end train the model first convert our listofstrings data to numpy arrays of integer indices the arrays are rightpadded we use categorical crossentropy as our loss since we re doing softmax classification moreover we use sparsecategoricalcrossentropy since our labels are integers export an endtoend model now we may want to export a model object that takes as input a string of arbitrary length rather than a sequence of indices it would make the model much more portable since you wouldn t have to worry about the input preprocessing pipeline our vectorizer is actually a keras layer so it s simple setup only the tensorflow backend supports string inputs introduction in this example we show how to train a text classification model that uses pre trained word embeddings we ll work with the newsgroup20 dataset a set of 20 000 message board messages belonging to 20 different topic categories for the pre trained word embeddings we ll use glove embeddings http nlp stanford edu projects glove download the newsgroup20 data let s take a look at the data here s a example of what one file contains as you can see there are header lines that are leaking the file s category either explicitly the first line is literally the category name or implicitly e g via the organization filed let s get rid of the headers there s actually one category that doesn t have the expected number of files but the difference is small enough that the problem remains a balanced classification problem shuffle and split the data into training validation sets shuffle the data extract a training validation split create a vocabulary index let s use the textvectorization to index the vocabulary found in the dataset later we ll use the same layer instance to vectorize the samples our layer will only consider the top 20 000 words and will truncate or pad sequences to be actually 200 tokens long you can retrieve the computed vocabulary used via vectorizer get_vocabulary let s print the top 5 words let s vectorize a test sentence as you can see the gets represented as 2 why not 0 given that the was the first word in the vocabulary that s because index 0 is reserved for padding and index 1 is reserved for out of vocabulary tokens here s a dict mapping words to their indices as you can see we obtain the same encoding as above for our test sentence load pre trained word embeddings let s download pre trained glove embeddings a 822m zip file you ll need to run the following commands shell wget https downloads cs stanford edu nlp data glove 6b zip unzip q glove 6b zip the archive contains text encoded vectors of various sizes 50 dimensional 100 dimensional 200 dimensional 300 dimensional we ll use the 100d ones let s make a dict mapping words strings to their numpy vector representation now let s prepare a corresponding embedding matrix that we can use in a keras embedding layer it s a simple numpy matrix where entry at index i is the pre trained vector for the word of index i in our vectorizer s vocabulary prepare embedding matrix words not found in embedding index will be all zeros this includes the representation for padding and oov next we load the pre trained word embeddings matrix into an embedding layer note that we set trainable false so as to keep the embeddings fixed we don t want to update them during training build the model a simple 1d convnet with global max pooling and a classifier at the end train the model first convert our list of strings data to numpy arrays of integer indices the arrays are right padded we use categorical crossentropy as our loss since we re doing softmax classification moreover we use sparse_categorical_crossentropy since our labels are integers export an end to end model now we may want to export a model object that takes as input a string of arbitrary length rather than a sequence of indices it would make the model much more portable since you wouldn t have to worry about the input preprocessing pipeline our vectorizer is actually a keras layer so it s simple
import os


os.environ["KERAS_BACKEND"] = "tensorflow"

import pathlib
import numpy as np
import tensorflow.data as tf_data
import keras
from keras import layers





data_path = keras.utils.get_file(
    "news20.tar.gz",
    "http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/news20.tar.gz",
    untar=True,
)



data_dir = pathlib.Path(data_path).parent / "20_newsgroup"
dirnames = os.listdir(data_dir)
print("Number of directories:", len(dirnames))
print("Directory names:", dirnames)

fnames = os.listdir(data_dir / "comp.graphics")
print("Number of files in comp.graphics:", len(fnames))
print("Some example filenames:", fnames[:5])



print(open(data_dir / "comp.graphics" / "38987").read())



samples = []
labels = []
class_names = []
class_index = 0
for dirname in sorted(os.listdir(data_dir)):
    class_names.append(dirname)
    dirpath = data_dir / dirname
    fnames = os.listdir(dirpath)
    print("Processing %s, %d files found" % (dirname, len(fnames)))
    for fname in fnames:
        fpath = dirpath / fname
        f = open(fpath, encoding="latin-1")
        content = f.read()
        lines = content.split("\n")
        lines = lines[10:]
        content = "\n".join(lines)
        samples.append(content)
        labels.append(class_index)
    class_index += 1

print("Classes:", class_names)
print("Number of samples:", len(samples))






seed = 1337
rng = np.random.RandomState(seed)
rng.shuffle(samples)
rng = np.random.RandomState(seed)
rng.shuffle(labels)


validation_split = 0.2
num_validation_samples = int(validation_split * len(samples))
train_samples = samples[:-num_validation_samples]
val_samples = samples[-num_validation_samples:]
train_labels = labels[:-num_validation_samples]
val_labels = labels[-num_validation_samples:]



vectorizer = layers.TextVectorization(max_tokens=20000, output_sequence_length=200)
text_ds = tf_data.Dataset.from_tensor_slices(train_samples).batch(128)
vectorizer.adapt(text_ds)



vectorizer.get_vocabulary()[:5]



output = vectorizer([["the cat sat on the mat"]])
output.numpy()[0, :6]



voc = vectorizer.get_vocabulary()
word_index = dict(zip(voc, range(len(voc))))



test = ["the", "cat", "sat", "on", "the", "mat"]
[word_index[w] for w in test]









path_to_glove_file = "glove.6B.100d.txt"

embeddings_index = {}
with open(path_to_glove_file) as f:
    for line in f:
        word, coefs = line.split(maxsplit=1)
        coefs = np.fromstring(coefs, "f", sep=" ")
        embeddings_index[word] = coefs

print("Found %s word vectors." % len(embeddings_index))



num_tokens = len(voc) + 2
embedding_dim = 100
hits = 0
misses = 0


embedding_matrix = np.zeros((num_tokens, embedding_dim))
for word, i in word_index.items():
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        
        
        embedding_matrix[i] = embedding_vector
        hits += 1
    else:
        misses += 1
print("Converted %d words (%d misses)" % (hits, misses))




from keras.layers import Embedding

embedding_layer = Embedding(
    num_tokens,
    embedding_dim,
    trainable=False,
)
embedding_layer.build((1,))
embedding_layer.set_weights([embedding_matrix])



int_sequences_input = keras.Input(shape=(None,), dtype="int32")
embedded_sequences = embedding_layer(int_sequences_input)
x = layers.Conv1D(128, 5, activation="relu")(embedded_sequences)
x = layers.MaxPooling1D(5)(x)
x = layers.Conv1D(128, 5, activation="relu")(x)
x = layers.MaxPooling1D(5)(x)
x = layers.Conv1D(128, 5, activation="relu")(x)
x = layers.GlobalMaxPooling1D()(x)
x = layers.Dense(128, activation="relu")(x)
x = layers.Dropout(0.5)(x)
preds = layers.Dense(len(class_names), activation="softmax")(x)
model = keras.Model(int_sequences_input, preds)
model.summary()



x_train = vectorizer(np.array([[s] for s in train_samples])).numpy()
x_val = vectorizer(np.array([[s] for s in val_samples])).numpy()

y_train = np.array(train_labels)
y_val = np.array(val_labels)



model.compile(
    loss="sparse_categorical_crossentropy", optimizer="rmsprop", metrics=["acc"]
)
model.fit(
    x_train, y_train, batch_size=128, epochs=20, validation_data=(x_val, y_val)
)



string_input = keras.Input(shape=(1,), dtype="string")
x = vectorizer(string_input)
preds = model(x)
end_to_end_model = keras.Model(string_input, preds)

probabilities = end_to_end_model(
    keras.ops.convert_to_tensor(
        [["this message is about computer graphics and 3D modeling"]]
    )
)

print(class_names[np.argmax(probabilities[0])])