<jupyter_start><jupyter_text>Load Text We will use lower-level utilities like `tf.data.TextLineDataset` to load text files, and `tf.text` to preprocess the data for finer-grain control.<jupyter_code>!pip install tensorflow_text -q
import collections
import pathlib
import re
import string

import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow_text as tf_text

from tensorflow.keras import layers
from tensorflow.keras import losses
from tensorflow.keras import preprocessing
from tensorflow.keras import utils
from tensorflow.keras import utils
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization

tf.__version__, tfds.__version__<jupyter_output><empty_output><jupyter_text>Predict the tag for a Stack Overflow question<jupyter_code>data_url = 'https://storage.googleapis.com/download.tensorflow.org/data/stack_overflow_16k.tar.gz'
dataset = utils.get_file(
    'stack_overflow_16k.tar.gz',
    data_url,
    untar=True,
    cache_dir='stack_overflow',
    cache_subdir=''
)
dataset_dir = pathlib.Path(dataset).parent
dataset_dir, pathlib.Path(dataset)
list(dataset_dir.iterdir())
train_dir = dataset_dir/'train'
list(train_dir.iterdir())<jupyter_output><empty_output><jupyter_text>The `train/csharp`, `train/java`, `train/python`, and `train/javascript` directories contain many text files, each of wich is Stack Overflow question. Print a file and inspect the data.<jupyter_code>sample_file = train_dir/'python/1757.txt'
with open(sample_file) as f:
  print(f.read())<jupyter_output>"i want to return a specific list when i have a collection of lists i want to return a specific list, when i have a collection of lists, but i am not to sure how to do this. i tried this approach but it didn't work. any ideas..this_list1 = [2,3,4,5].this_list2 = [5,6,9,8]..x = input(""which list do you want"")..print this_list(x)"<jupyter_text>Load the datasetNext, we will load the data off disk and prepare it into a format suitable for training. To do so, you will use `text_dataset_from_directory` utility to create a labeled `tf.data.Dataset`.The `preprocessing.text_dataset_from_directory` expects a directory structure as follows.```train/...csharp/......1.txt......2.txt...java/......1.txt......2.txt...javascript/......1.txt......2.txt...python/......1.txt......2.txt When running a machine learning experiment, it is a best practice to divide your dataset into three splits: train, validation and test. Test Stack Overflow has already been divided into train and test, but it lack a validation set. Create a validation set using a 80:20 split of the training data by using the `validation_split` argument below.<jupyter_code>batch_size = 32
seed = 42

raw_train_ds = preprocessing.text_dataset_from_directory(
    train_dir,
    batch_size=batch_size,
    validation_split=0.2,
    subset='training',
    seed=seed    
)<jupyter_output>Found 8000 files belonging to 4 classes.
Using 6400 files for training.<jupyter_text>As you can see above, there are 8,000 examples in the training folder, of which you will use 80% (or 6,400) for training. As you will see in a moment, you can train a model by passing a `tf.data.Dataset` directly to `model.fit`. First, iterate over the dataset and print out a few examples, to get a feel for the data.Note: To increase the difficulty of the classification problem, the dataset author replaced occurrences of the words *Python*, *CSharp*, *JavaScript*, or *Java* in the programming question with the word *blank*.<jupyter_code>for text_batch, label_batch in raw_train_ds.take(1):
  for i in range(10):
    print('Question:', text_batch.numpy()[i][:100], '...')
    print('Label:', label_batch.numpy()[i])<jupyter_output>Question: b'"my tester is going to the wrong constructor i am new to programming so if i ask a question that can' ...
Label: 1
Question: b'"blank code slow skin detection this code changes the color space to lab and using a threshold finds' ...
Label: 3
Question: b'"option and validation in blank i want to add a new option on my system where i want to add two text' ...
Label: 1
Question: b'"exception: dynamic sql generation for the updatecommand is not supported against a selectcommand th' ...
Label: 0
Question: b'"parameter with question mark and super in blank, i\'ve come across a method that is formatted like t' ...
Label: 1
Question: b'call two objects wsdl the first time i got a very strange wsdl. ..i would like to call the object (i' ...
Label: 0
Question: b'how to correctly make the icon for systemtray in blank using icon sizes of any dimension for systemt' ...
Label: 0
Question: b'"is there a way to check a variable that exists in a different script than the original one? i\'m [...]<jupyter_text>The labels are `0`, `1`, `2` or `3`. To see which of these correspond to which string label, you can check the `class_names` property on the dataset.<jupyter_code>for i, label in enumerate(raw_train_ds.class_names):
  print('Label', i, 'corresponds to', label)
raw_train_ds.class_names
raw_val_ds = preprocessing.text_dataset_from_directory(
    train_dir,
    batch_size=batch_size,
    validation_split=0.2,
    subset='validation',
    seed=seed
)
test_dir = dataset_dir/'test'
raw_test_ds = preprocessing.text_dataset_from_directory(
    test_dir,
    batch_size=batch_size
)<jupyter_output>Found 8000 files belonging to 4 classes.<jupyter_text>Next, you will standardize, tokenize, and vectorize the data using the `preprocessing.TextVectorization` layer.* Standardization refers to preprocessing the text, typically to remove punctuation or HTML elements to simplify the dataset.* Tokenization refers to splitting strings into tokens (for example, splitting a sentence into individual words by splitting on whitespace).* Vectorization refers to converting tokens into numbers so they can be fed into a neural network.All of these tasks can be accomplished with this layer. You can learn more about each of these in the [API doc](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/TextVectorization).* The default standardization converts text to lowercase and removes punctuation.* The default tokenizer splits on whitespace.* The default vectorization mode is `int`. This outputs integer indices (one per token). This mode can be used to build models that take word order into account. You can also use other modes, like `binary`, to build bag-of-word models.You will build two modes to learn more about these. First, you will use the `binary` model to build a bag-of-words model. Next, you will use the `int` mode with a 1D ConvNet.<jupyter_code>VOCAB_SIZE=10000

binary_vectorize_layer = TextVectorization(
    max_tokens=VOCAB_SIZE,
    output_mode='binary'
)<jupyter_output><empty_output><jupyter_text>For `int` mode, in addition to maximum vocabulary size, you need to set an explicit maximum sequence length, which will cause the layer to pad or truncate sequences to exactly sequence_length values.<jupyter_code>MAX_SEQUENCE_LENGTH = 250

int_vectorize_layer = TextVectorization(
    max_tokens=VOCAB_SIZE,
    output_mode='int',
    output_sequence_length=MAX_SEQUENCE_LENGTH
)<jupyter_output><empty_output><jupyter_text>Next, you will call adapt to fit the state of the preprocessing layer to the dataset. This will cause the model to build an index of strings to integers.Note: it's important to only use your training data when calling adapt (using the test set would leak information).<jupyter_code># make a text-only dataset (without labels), then call adapt
train_text = raw_train_ds.map(lambda text, labels: text)
binary_vectorize_layer.adapt(train_text)
int_vectorize_layer.adapt(train_text)<jupyter_output><empty_output><jupyter_text>See the results of using these layers to process data:<jupyter_code>def binary_vectorize_text(text, label):
  text = tf.expand_dims(text, -1)
  return binary_vectorize_layer(text), label
def int_vectorize_text(text, label):
  text = tf.expand_dims(text, -1)
  return int_vectorize_layer(text), label
# retrieve a batch (of 32 reviews and labels) from the dataset
text_batch, label_batch = next(iter(raw_train_ds))
first_question, first_label = text_batch[0], label_batch[0]
print('question:', first_question)
print('label:', first_label)
print('binary vectorized question:', binary_vectorize_text(first_question, first_label)[0])
print('int vectorized question:', int_vectorize_text(first_question, first_label)[0])<jupyter_output>int vectorized question: tf.Tensor(
[[  38  450   65    7   16   12  892  265  186  451   44   11    6  685
     3   46    4 2062    2  485    1    6  158    7  479    1   26   20
   158    7  479    1  502   38  450    1 1767 1763    1    1    1    1
     1    1    1    1    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0  [...]<jupyter_text>As you can see above, `binary` mode returns an array denoting which tokens exist at least once in the input, while `int` mode replaces each token by an integer, thus preserving their order. You can lookup the token (string) that each integer corresponds to by calling `.get_vocabulary()` on the layer.<jupyter_code>print('1289 --->', int_vectorize_layer.get_vocabulary()[1289])
print('2200 --->', int_vectorize_layer.get_vocabulary()[2200])
print(f'Vocabulary size: {len(int_vectorize_layer.get_vocabulary())}')<jupyter_output>1289 ---> roman
2200 ---> desktop
Vocabulary size: 10000<jupyter_text>You are nearly ready to train your model. As a final preprocessing step, you will apply the `TextVectorization` layers you created earlier to the train, validation, and test dataset.<jupyter_code>binary_train_ds = raw_train_ds.map(binary_vectorize_text)
binary_val_ds = raw_val_ds.map(binary_vectorize_text)
binary_test_ds = raw_test_ds.map(binary_vectorize_text)

int_train_ds = raw_train_ds.map(int_vectorize_text)
int_val_ds = raw_val_ds.map(int_vectorize_text)
int_test_ds = raw_test_ds.map(int_vectorize_text)<jupyter_output><empty_output><jupyter_text>Configure the dataset for performanceThese are two important methods you should use when loading data to make sure that I/O does not become blocking.`.cache()` keeps data in memory after it's loaded off disk. This will ensure the dataset does not become a bottleneck while training your model. If your dataset is too large to fit into memory, you can also use this method to create a performant on-disk cache, which is more efficient to read than many small files.`.prefetch()` overlaps data preprocessing and model execution while training.<jupyter_code>AUTOTUNE = tf.data.experimental.AUTOTUNE

def configure_dataset(dataset):
  return dataset.cache().prefetch(buffer_size=AUTOTUNE)
binary_train_ds = configure_dataset(binary_train_ds)
binary_val_ds = configure_dataset(binary_val_ds)
binary_test_ds = configure_dataset(binary_test_ds)

int_train_ds = configure_dataset(int_train_ds)
int_val_ds = configure_dataset(int_val_ds)
int_test_ds = configure_dataset(int_test_ds)
binary_model = tf.keras.Sequential([layers.Dense(4)])
binary_model.compile(
    loss=losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy']
)
history = binary_model.fit(
    binary_train_ds,
    validation_data=binary_val_ds,
    epochs=10
)
def create_model(vocab_size, num_labels):
  model = tf.keras.Sequential([
    layers.Embedding(vocab_size, 64, mask_zero=True),
    layers.Conv1D(64, 5, padding='valid', activation='relu', strides=2),
    layers.GlobalMaxPool1D(),
    layers.Dense(num_labels)
  ])
  return model
int_model = create_model(vocab_size=VOCAB_SIZE + 1, num_labels=4)
int_model.compile(
    loss=losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy']
)
history = int_model.fit(int_train_ds, validation_data=int_val_ds, epochs=5)
print('Linear model on binary vectorized data:')
print(binary_model.summary())
print('ConvNet model on int vectorized data:')
print(int_model.summary())
binary_loss, binary_accuracy = binary_model.evaluate(binary_test_ds)
int_loss, int_accuracy = int_model.evaluate(int_test_ds)

print(f'Binary model accuracy: {binary_accuracy:2.2%}')
print(f'Int model accuracy: {int_accuracy:2.2%}')<jupyter_output>250/250 [==============================] - 2s 9ms/step - loss: 0.5170 - accuracy: 0.8141
250/250 [==============================] - 2s 9ms/step - loss: 0.5147 - accuracy: 0.8116
Binary model accuracy: 81.41%
Int model accuracy: 81.16%<jupyter_text>Export the modelIn the code above, you applied the `TextVectorization` layer to the dataset before feeding text to the model. If you want to make your model capable of processing raw strings (for example, to simplify deploying it), you can include the `TextVectorization` layer inside your model. To do so, you can create a new model using the weights you just trained.<jupyter_code>export_model = tf.keras.Sequential([
  binary_vectorize_layer,
  binary_model,
  layers.Activation('sigmoid')
])

export_model.compile(
    loss=losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy']
)

#export_model.build()
#export_model.summary()<jupyter_output><empty_output><jupyter_text>Now your model can take raw strings as input and predict a score for each label using model.predict. Define a function to find the label with the maximum score:<jupyter_code>def get_string_labels(predicted_scores_batch):
  predicted_int_labels = tf.argmax(predicted_scores_batch, axis=1)
  predicted_labels = tf.gather(
      raw_train_ds.class_names, 
      predicted_int_labels
  )
  return predicted_labels<jupyter_output><empty_output><jupyter_text>Run inference on new data<jupyter_code>inputs = [
  "how do I extract keys from a dict into a list?",  # python
  "debug public static void main(string[] args) {...}",  # java
]

predicted_scores = export_model.predict(inputs)
predicted_labels = get_string_labels(predicted_scores)
for input, label in zip(inputs, predicted_labels):
  print("Question: ", input)
  print("Predicted label: ", label.numpy())<jupyter_output>Question:  how do I extract keys from a dict into a list?
Predicted label:  b'python'
Question:  debug public static void main(string[] args) {...}
Predicted label:  b'java'<jupyter_text>Including the text preprocessing logic inside your model enables you to export a model for production that simplifies deployment, and reduces the potential for [train/test skew](https://developers.google.com/machine-learning/guides/rules-of-mltraining-serving_skew).> Rule 32: Re-use code between your training pipeline and your serving pipeline whenever possible.There is a performance difference to keep in mind when choosing where to apply your `TextVectorization` layer. Using it outside of your model enables you to do asynchronous CPU processing and buffering of your data when training on GPU. So, if you're training your model on the GPU, you probably want to go with this option to get the best performance while developing your model, then switch to including the TextVectorization layer inside your model when you're ready to prepare for deployment.> Using `TextVectorization` outside of your model enables you to do asynchronous CPU processing and buffering of your data when training on GPU.  Predict the author of Illiad translations The following provides an example of using `tf.data.TextLineDataset` to load examples from text files, and `tf.text` to preprocess the data. In this example, you will use three different English translations of the same work, Homer's Illiad, and train a model to identify the translator given a single line of text.  Download and explore the datasetThe texts of the three translations are by: - [<NAME>](https://en.wikipedia.org/wiki/William_Cowper) — [text](https://storage.googleapis.com/download.tensorflow.org/data/illiad/cowper.txt) - [<NAME>](https://en.wikipedia.org/wiki/Edward_Smith-Stanley,_14th_Earl_of_Derby) — [text](https://storage.googleapis.com/download.tensorflow.org/data/illiad/derby.txt)- [<NAME>](https://en.wikipedia.org/wiki/Samuel_Butler_%28novelist%29) — [text](https://storage.googleapis.com/download.tensorflow.org/data/illiad/butler.txt)The text files used in this tutorial have undergone some typical preprocessing tasks like removing document header and footer, line numbers and chapter titles. Download these lightly munged files locally.<jupyter_code>DIRECTORY_URL = 'https://storage.googleapis.com/download.tensorflow.org/data/illiad/'
FILE_NAMES = ['cowper.txt', 'derby.txt', 'butler.txt']

for name in FILE_NAMES:
  text_dir = utils.get_file(name, origin=DIRECTORY_URL + name)

parent_dir = pathlib.Path(text_dir).parent
list(parent_dir.iterdir())<jupyter_output><empty_output><jupyter_text>Load the datasetYou will use `TextLineDataset`, which is designed to create a `tf.data.Dataset` from a text file in which each example is a line of text from the original file, whereas `text_dataset_from_directory` treats all contents of a file as a single example. `TextLineDataset` is useful for text data that is primarily line-based (for example, poetry or error logs). > `TextLineDataset` reads each sample from a line of text from the original file, whereas `text_dataset_from_directory` treats all contents of as a single sample.Iterate through these files, loading each one into its own dataset. Each example needs to be individually labeled, so use `tf.data.Dataset.map` to apply a labeler function to each one. This will iterate over every example in the dataset, returning (`example, label`) pairs.<jupyter_code>def labeler(example, index):
  return example, tf.cast(index, tf.int64)
labeled_data_sets = []

for i, file_name in enumerate(FILE_NAMES):
  lines_dataset = tf.data.TextLineDataset(str(parent_dir/file_name))
  labeled_dataset = lines_dataset.map(lambda ex: labeler(ex, i))
  labeled_data_sets.append(labeled_dataset)

labeled_data_sets[:3]<jupyter_output><empty_output><jupyter_text>Next, you'll combine these labeled datasets into a single dataset, and shuffle it.<jupyter_code>BUFFER_SIZE = 50000
BATCH_SIZE = 64
VALIDATION_SIZE = 5000
all_labeled_data = labeled_data_sets[0]
for labeled_dataset in labeled_data_sets[1:]:
  all_labeled_data = all_labeled_data.concatenate(labeled_dataset)

all_labeled_data = all_labeled_data.shuffle(
    BUFFER_SIZE, 
    reshuffle_each_iteration=False
)

tf.data.experimental.cardinality(all_labeled_data)<jupyter_output><empty_output><jupyter_text>Print out a few examples as before. The dataset hasn't been batched yet, hence each entry in all_labeled_data corresponds to one data point<jupyter_code>for text, label in all_labeled_data.take(10):
  print('Sentence:', text.numpy())
  print('Label:', label.numpy())
text, label = next(all_labeled_data.as_numpy_iterator())
text, label
text, label = next(iter(all_labeled_data.take(1)))
text.numpy(), label.numpy()<jupyter_output><empty_output><jupyter_text>Prepare the dataset for trainingInstead of using the Keras `TextVectorization` layer to preprocess our text dataset, you will now use the [`tf.text` API](https://www.tensorflow.org/tutorials/tensorflow_text/intro) to standardize and tokenize the data, build a vocabulary and use `StaticVocabularyTable` to map tokens to integers to feed to the model.While tf.text provides various tokenizers, you will use the `UnicodeScriptTokenizer` to tokenize our dataset. Define a function to convert the text to lower-case and tokenize it. You will use `tf.data.Dataset.map` to apply the tokenization to the dataset.<jupyter_code>tokenizer = tf_text.UnicodeScriptTokenizer()
def tokenize(text, unused_label):
  lower_case = tf_text.case_fold_utf8(text)
  return tokenizer.tokenize(lower_case)
tokenized_ds = all_labeled_data.map(tokenize)<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201: batch_gather (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2017-10-25.
Instructions for updating:
`tf.batch_gather` is deprecated, please use `tf.gather` with `batch_dims=-1` instead.<jupyter_text>Next, you will build a vocabulary by sorting tokens by frequency and keeping the top `VOCAB_SIZE` tokens.<jupyter_code>tokenized_ds = configure_dataset(tokenized_ds)

vocab_dict = collections.defaultdict(lambda: 0)
for toks in tokenized_ds.as_numpy_iterator():
  for tok in toks:
    vocab_dict[tok] += 1

sorted(list(vocab_dict.items()), key=lambda x: x[1], reverse=True)[:10]
sorted(list(vocab_dict.items()))[:10]
vocab = sorted(vocab_dict.items(), key=lambda x:x[1], reverse=True)
vocab = [token for token, count in vocab]
vocab = vocab[:VOCAB_SIZE]
vocab_size = len(vocab)

print('Vocab size:', vocab_size)
print('First five vocab entries:', vocab[:5])<jupyter_output>Vocab size: 10000
First five vocab entries: [b',', b'the', b'and', b"'", b'of']<jupyter_text>To convert the tokens into integers, use the `vocab` set to create a `StaticVocabularyTable`. You will map tokens to integers in the range [`2`, `vocab_size + 2`]. As with the `TextVectorization` layer, `0` is reserved to denote padding and `1` is reserved to denote an out-of-vocabulary (OOV) token.<jupyter_code>keys = vocab
values = range(2, len(vocab) + 2)

init = tf.lookup.KeyValueTensorInitializer(
    keys, values, key_dtype=tf.string, value_dtype=tf.int64
)

num_oov_buckets = 1
vocab_table = tf.lookup.StaticVocabularyTable(init, num_oov_buckets)<jupyter_output><empty_output><jupyter_text>Finally, define a function to standarize, tokenize and vectorize the dataset using the tokenizer and lookup table:<jupyter_code>def preprocess_text(text, label):
  standarized = tf_text.case_fold_utf8(text)
  tokenized = tokenizer.tokenize(standarized)
  vectorized = vocab_table.lookup(tokenized)
  return vectorized, label<jupyter_output><empty_output><jupyter_text>You can try this on a single example to see the output.<jupyter_code>example_text, example_label = next(all_labeled_data.as_numpy_iterator())
print('Sentence:', example_text)

vectorized_text, example_label = preprocess_text(example_text, example_label)
print('Vectorized sentence:', vectorized_text)<jupyter_output>Sentence: b'scales and placed a doom in each of them, one for Achilles and the'
Vectorized sentence: tf.Tensor(
[2175    4  836   15  435   13  148    6   39    2   79   19   57    4
    3], shape=(15,), dtype=int64)<jupyter_text>Now run the preprocess function on the dataset using tf.data.Dataset.map<jupyter_code>all_encoded_data = all_labeled_data.map(preprocess_text)<jupyter_output><empty_output><jupyter_text>Split the dataset into train and testThe Keras `TextVectorization` layer also batches and pads the vectorized data. Padding is required because the examples inside of a batch need to be the same size and shape, but the examples in these datasets are not all the same size — each line of text has a different number of words. `tf.data.Dataset` supports splitting and padded-batching datasets:<jupyter_code>train_data = all_encoded_data.skip(VALIDATION_SIZE).shuffle(BUFFER_SIZE)
validation_data = all_encoded_data.take(VALIDATION_SIZE)
train_data = train_data.padded_batch(BATCH_SIZE)
validation_data = validation_data.padded_batch(BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>Now, `validation_data` and `train_data` are not collections of (`example, label`) pairs, but collections of batches. Each batch is a pair of (*many examples*, *many labels*) represented as arrays. To illustrate:<jupyter_code>sample_text, sample_labels = next(validation_data.as_numpy_iterator())

print('Text batch shape:', sample_text.shape)
print('Label batch shape:', sample_labels.shape)
print('First text exmaple:', sample_text[0])
print('First label example:', sample_labels[0])<jupyter_output>Text batch shape: (64, 17)
Label batch shape: (64,)
First text exmaple: [2175    4  836   15  435   13  148    6   39    2   79   19   57    4
    3    0    0]
First label example: 2<jupyter_text>Since we use `0` for padding and `1` for out-of-vocabulary (OOV) tokens, the vocabulary has increased by two.<jupyter_code>vocab_size += 2<jupyter_output><empty_output><jupyter_text>Configure the datasets for better performance as before.<jupyter_code>train_data = configure_dataset(train_data)
validation_data = configure_dataset(validation_data)<jupyter_output><empty_output><jupyter_text>Train the modelYou can train a model on this dataset as before.<jupyter_code>model = create_model(vocab_size=vocab_size, num_labels=3)
model.compile(
    optimizer='adam',
    loss=losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy']
)
history = model.fit(train_data, validation_data=validation_data, epochs=3)
loss, accuracy = model.evaluate(validation_data)
print('Loss:', loss)
print(f'Accuracy: {accuracy:2.2%}')
preprocess_layer = TextVectorization(
    max_tokens=vocab_size,
    standardize=tf_text.case_fold_utf8,
    split=tokenizer.tokenize,
    output_mode='int',
    output_sequence_length=MAX_SEQUENCE_LENGTH)
preprocess_layer.set_vocabulary(vocab)
export_model = tf.keras.Sequential(
    [preprocess_layer, model,
     layers.Activation('sigmoid')])

export_model.compile(
    loss=losses.SparseCategoricalCrossentropy(from_logits=False),
    optimizer='adam',
    metrics=['accuracy'])
# Create a test dataset of raw strings
test_ds = all_labeled_data.take(VALIDATION_SIZE).batch(BATCH_SIZE)
test_ds = configure_dataset(test_ds)
loss, accuracy = export_model.evaluate(test_ds)
print("Loss: ", loss)
print("Accuracy: {:2.2%}".format(accuracy))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09071439498240783, 'fraction_numerical': 0.021837234205292946, 'mean_word_length': 4.985122453650721, 'pattern_counts': {'":': 0, '<': 116, '<?xml version=': 0, '>': 123, 'https://': 11, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '625731', 'license': 'Apache-2.0', 'n_tokens_mistral': 8402, 'n_tokens_neox': 8037, 'n_words': 2670, 'path': 'deeplearning.ai/tf/c3_w2_loading_text.ipynb', 'repo_name': 'martin-fabbri/colab-notebooks', 'size': '26178'}