title text classification from scratch s mark omernick francois chollet date created 20191106 last modified 20200517 description text sentiment classification starting from raw text files accelerator gpu introduction this example shows how to do text classification starting from raw text as a set of text files on disk we demonstrate the workflow on the imdb sentiment classification dataset unprocessed version we use the textvectorization layer for word splitting indexing setup load the data imdb movie review sentiment classification let s download the data and inspect its structure the aclimdb folder contains a train and test subfolder the aclimdbtrainpos and aclimdbtrainneg folders contain text files each of which represents one review either positive or negative we are only interested in the pos and neg subfolders so let s delete the rest you can use the utility keras utils textdatasetfromdirectory to generate a labeled tf data dataset object from a set of text files on disk filed into classspecific folders let s use it to generate the training validation and test datasets the validation and training datasets are generated from two subsets of the train directory with 20 of samples going to the validation dataset and 80 going to the training dataset having a validation dataset in addition to the test dataset is useful for tuning hyperparameters such as the model architecture for which the test dataset should not be used before putting the model out into the real world however it should be retrained using all available training data without creating a validation dataset so its performance is maximized when using the validationsplit subset arguments make sure to either specify a random seed or to pass shufflefalse so that the validation training splits you get have no overlap let s preview a few samples it s important to take a look at your raw data to ensure your normalization and tokenization will work as expected we can do that by taking a few examples from the training set and looking at them this is one of the places where eager execution shines we can just evaluate these tensors using numpy instead of needing to evaluate them in a sessiongraph context prepare the data in particular we remove br tags having looked at our data above we see that the raw text contains html break tags of the form br these tags will not be removed by the default standardizer which doesn t strip html because of this we will need to create a custom standardization function model constants now that we have our custom standardization we can instantiate our text vectorization layer we are using this layer to normalize split and map strings to integers so we set our outputmode to int note that we re using the default split function and the custom standardization defined above we also set an explicit maximum sequence length since the cnns later in our model won t support ragged sequences now that the vocab layer has been created call adapt on a textonly dataset to create the vocabulary you don t have to batch but for very large datasets this means you re not keeping spare copies of the dataset in memory let s make a textonly dataset no labels let s call adapt two options to vectorize the data there are 2 ways we can use our text vectorization layer option 1 make it part of the model so as to obtain a model that processes raw strings like this python textinput keras inputshape1 dtypetf string name text x vectorizelayertextinput x layers embeddingmaxfeatures 1 embeddingdimx option 2 apply it to the text dataset to obtain a dataset of word indices then feed it into a model that expects integer sequences as inputs an important difference between the two is that option 2 enables you to do asynchronous cpu processing and buffering of your data when training on gpu so if you re training the model on gpu you probably want to go with this option to get the best performance this is what we will do below if we were to export our model to production we d ship a model that accepts raw strings as input like in the code snippet for option 1 above this can be done after training we do this in the last section vectorize the data do async prefetching buffering of the data for best performance on gpu build a model we choose a simple 1d convnet starting with an embedding layer a integer input for vocab indices next we add a layer to map those vocab indices into a space of dimensionality embeddingdim conv1d global max pooling we add a vanilla hidden layer we project onto a single unit output layer and squash it with a sigmoid train the model fit the model using the train and test datasets evaluate the model on the test set make an endtoend model if you want to obtain a model capable of processing raw strings you can simply create a new model using the weights we just trained a string input turn strings into vocab indices turn vocab indices into predictions our end to end model test it with rawtestds which yields raw strings introduction this example shows how to do text classification starting from raw text as a set of text files on disk we demonstrate the workflow on the imdb sentiment classification dataset unprocessed version we use the textvectorization layer for word splitting indexing setup load the data imdb movie review sentiment classification let s download the data and inspect its structure the aclimdb folder contains a train and test subfolder the aclimdb train pos and aclimdb train neg folders contain text files each of which represents one review either positive or negative we are only interested in the pos and neg subfolders so let s delete the rest you can use the utility keras utils text_dataset_from_directory to generate a labeled tf data dataset object from a set of text files on disk filed into class specific folders let s use it to generate the training validation and test datasets the validation and training datasets are generated from two subsets of the train directory with 20 of samples going to the validation dataset and 80 going to the training dataset having a validation dataset in addition to the test dataset is useful for tuning hyperparameters such as the model architecture for which the test dataset should not be used before putting the model out into the real world however it should be retrained using all available training data without creating a validation dataset so its performance is maximized when using the validation_split subset arguments make sure to either specify a random seed or to pass shuffle false so that the validation training splits you get have no overlap let s preview a few samples it s important to take a look at your raw data to ensure your normalization and tokenization will work as expected we can do that by taking a few examples from the training set and looking at them this is one of the places where eager execution shines we can just evaluate these tensors using numpy instead of needing to evaluate them in a session graph context prepare the data in particular we remove br tags having looked at our data above we see that the raw text contains html break tags of the form br these tags will not be removed by the default standardizer which doesn t strip html because of this we will need to create a custom standardization function model constants now that we have our custom standardization we can instantiate our text vectorization layer we are using this layer to normalize split and map strings to integers so we set our output_mode to int note that we re using the default split function and the custom standardization defined above we also set an explicit maximum sequence length since the cnns later in our model won t support ragged sequences now that the vocab layer has been created call adapt on a text only dataset to create the vocabulary you don t have to batch but for very large datasets this means you re not keeping spare copies of the dataset in memory let s make a text only dataset no labels let s call adapt two options to vectorize the data there are 2 ways we can use our text vectorization layer option 1 make it part of the model so as to obtain a model that processes raw strings like this python text_input keras input shape 1 dtype tf string name text x vectorize_layer text_input x layers embedding max_features 1 embedding_dim x option 2 apply it to the text dataset to obtain a dataset of word indices then feed it into a model that expects integer sequences as inputs an important difference between the two is that option 2 enables you to do asynchronous cpu processing and buffering of your data when training on gpu so if you re training the model on gpu you probably want to go with this option to get the best performance this is what we will do below if we were to export our model to production we d ship a model that accepts raw strings as input like in the code snippet for option 1 above this can be done after training we do this in the last section vectorize the data do async prefetching buffering of the data for best performance on gpu build a model we choose a simple 1d convnet starting with an embedding layer a integer input for vocab indices next we add a layer to map those vocab indices into a space of dimensionality embedding_dim conv1d global max pooling we add a vanilla hidden layer we project onto a single unit output layer and squash it with a sigmoid train the model fit the model using the train and test datasets evaluate the model on the test set make an end to end model if you want to obtain a model capable of processing raw strings you can simply create a new model using the weights we just trained a string input turn strings into vocab indices turn vocab indices into predictions our end to end model test it with raw_test_ds which yields raw strings
import tensorflow as tf
import keras
from keras.layers import TextVectorization
from keras import layers
import string
import re
import os
from pathlib import Path



fpath = keras.utils.get_file(
    origin="https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
)
dirpath = Path(fpath).parent.absolute()
os.system(f"tar -xf {fpath} -C {dirpath}")




os.system(f"ls {dirpath}/aclImdb")
os.system(f"ls {dirpath}/aclImdb/train")
os.system(f"ls {dirpath}/aclImdb/test")



os.system(f"cat {dirpath}/aclImdb/train/pos/6248_7.txt")



os.system(f"rm -r {dirpath}/aclImdb/train/unsup")




batch_size = 32
raw_train_ds, raw_val_ds = keras.utils.text_dataset_from_directory(
    f"{dirpath}/aclImdb/train",
    batch_size=batch_size,
    validation_split=0.2,
    subset="both",
    seed=1337,
)
raw_test_ds = keras.utils.text_dataset_from_directory(
    f"{dirpath}/aclImdb/test", batch_size=batch_size
)

print(f"Number of batches in raw_train_ds: {raw_train_ds.cardinality()}")
print(f"Number of batches in raw_val_ds: {raw_val_ds.cardinality()}")
print(f"Number of batches in raw_test_ds: {raw_test_ds.cardinality()}")









for text_batch, label_batch in raw_train_ds.take(1):
    for i in range(5):
        print(text_batch.numpy()[i])
        print(label_batch.numpy()[i])








def custom_standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    stripped_html = tf.strings.regex_replace(lowercase, "<br />", " ")
    return tf.strings.regex_replace(
        stripped_html, f"[{re.escape(string.punctuation)}]", ""
    )



max_features = 20000
embedding_dim = 128
sequence_length = 500








vectorize_layer = TextVectorization(
    standardize=custom_standardization,
    max_tokens=max_features,
    output_mode="int",
    output_sequence_length=sequence_length,
)






text_ds = raw_train_ds.map(lambda x, y: x)

vectorize_layer.adapt(text_ds)






def vectorize_text(text, label):
    text = tf.expand_dims(text, -1)
    return vectorize_layer(text), label



train_ds = raw_train_ds.map(vectorize_text)
val_ds = raw_val_ds.map(vectorize_text)
test_ds = raw_test_ds.map(vectorize_text)


train_ds = train_ds.cache().prefetch(buffer_size=10)
val_ds = val_ds.cache().prefetch(buffer_size=10)
test_ds = test_ds.cache().prefetch(buffer_size=10)




inputs = keras.Input(shape=(sequence_length,), dtype="int64")


x = keras.layers.Embedding(max_features, embedding_dim)(inputs)
x = layers.Dropout(0.5)(x)


x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.GlobalMaxPooling1D()(x)


x = layers.Dense(128, activation="relu")(x)
x = layers.Dropout(0.5)(x)


predictions = layers.Dense(1, activation="sigmoid", name="predictions")(x)
model = keras.Model(inputs, predictions)
model.summary()
model.compile(
    loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"]
)




epochs = 3


model.fit(train_ds, validation_data=val_ds, epochs=epochs)



model.evaluate(test_ds)




inputs = keras.Input(shape=(1,), dtype="string")

indices = vectorize_layer(inputs)

outputs = model(indices)


end_to_end_model = keras.Model(inputs, outputs)
end_to_end_model.compile(
    loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"]
)


end_to_end_model.evaluate(raw_test_ds)