<jupyter_start><jupyter_text>Padding and Masking Padding - Adding zeros at end of the sequence to keep the sequence length constant<jupyter_code>text = [
  ["The", "weather", "will", "be", "nice", "tomorrow"],
  ["How", "are", "you", "doing", "today"],
  ["Hello", "world", "!"]
]
tokenzing = [
  [83, 91, 1, 645, 1253, 927],
  [73, 8, 3215, 55, 927],
  [71, 1331, 4231]
]

padding = [[  83   91    1  645 1253  927]
 [  73    8 3215   55  927    0]
 [ 711  632   71    0    0    0]]<jupyter_output><empty_output><jupyter_text>Masking- "Masking" is how layers are able to know when to skip / ignore certain timesteps in sequence inputs.- Some layers are mask-generators: Embedding can generate a mask from input values (if mask_zero=True), and so can the Masking layer.- Some layers are mask-consumers: they expose a mask argument in their __call__ method. This is the case for RNN layers.- In the Functional API and Sequential API, mask information is propagated automatically.- When writing subclassed models or when using layers in a standalone way, pass the mask arguments to layers manually.0 You can easily write layers that modify the current mask, that generate a new mask, or that consume the mask associated with the inputs.  Test classification<jupyter_code>%tensorflow_version 2.x
import tensorflow_datasets as tfds
import tensorflow as tf
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Load IMDB Data - IMDB data is already tokenzied so we dont need to tokenize it<jupyter_code>dataset, info = tfds.load('imdb_reviews/subwords8k', with_info=True,
                          as_supervised=True)
train_examples, test_examples = dataset['train'], dataset['test']

BUFFER_SIZE = 10000
BATCH_SIZE = 64<jupyter_output><empty_output><jupyter_text>- Note: As of TensorFlow 2.2 the padded_shapes argument is no longer required. The default behavior is to pad all axes to the longest in the batch.In tf v2.2 usetrain_dataset = (train_examples                 .shuffle(BUFFER_SIZE)                 .padded_batch(BATCH_SIZE))test_dataset = (test_examples                .padded_batch(BATCH_SIZE))<jupyter_code>train_dataset = (train_examples.shuffle(BUFFER_SIZE).padded_batch(BATCH_SIZE, padded_shapes=([None], [])))
test_dataset = (test_examples.padded_batch(BATCH_SIZE, padded_shapes=([None], [])))<jupyter_output><empty_output><jupyter_text>Build the model<jupyter_code>encoder = info.features['text'].encoder
model = tf.keras.Sequential()
model.add(tf.keras.layers.Embedding(encoder.vocab_size, 64))
model.add(tf.keras.layers.Bidirectional(tf.keras.layers.GRU(32, return_sequences=True, recurrent_dropout=0.2)))
model.add(tf.keras.layers.Bidirectional(tf.keras.layers.GRU(32, return_sequences=True, recurrent_dropout=0.2)))
model.add(tf.keras.layers.Dense(64, activation='relu'))
model.add(tf.keras.layers.Dense(1, activation='sigmoid'))
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(1e-4),
              metrics=['accuracy'])
model.summary()
history = model.fit(train_dataset, epochs=10,
                    validation_data=test_dataset, 
                    validation_steps=30)
test_loss, test_acc = model.evaluate(test_dataset)
print('Test Loss: {}'.format(test_loss))
print('Test Accuracy: {}'.format(test_acc))<jupyter_output><empty_output><jupyter_text>The above model does not mask the padding applied to the sequences. This can lead to skew if trained on padded sequences and test on un-padded sequences. Ideally you would use masking to avoid this, but as you can see below it only have a small effect on the output.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10657966286025014, 'fraction_numerical': 0.029635671560630777, 'mean_word_length': 4.757433489827856, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '615932', 'license': 'MIT', 'n_tokens_mistral': 1257, 'n_tokens_neox': 1161, 'n_words': 341, 'path': 'Learning_Tensorflow/Advanced_Tensorflow/text/Text_classification_RNN.ipynb', 'repo_name': 'oke-aditya/Machine_Learning', 'size': '3678'}