<jupyter_start><jupyter_text>6. Deep Learning for Text and Sequences  Sequence Processing with convnets Convnets have a particularly performance in computer vision problems due to their ability to operate convolutionally, extracting features from local input patches and allowing for representation modularity and data efficiency. These same properties allow them to be useful at sequence processing.1D convnets can be competitive with RNNs on certain sequence processing problems, usually at a considerably cheaper computational cost. Recently, 1D convnets, typically used with dilated kernals have been used with great success for audio generation and machine translation.<jupyter_code>import warnings
warnings.filterwarnings("ignore")

import pandas as pd
import matplotlib.pyplot as plt

from tensorflow.keras.datasets import imdb

from tensorflow.keras.backend import clear_session
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.layers import (Embedding, Dense, SimpleRNN, LSTM, MaxPooling1D, Conv1D, GlobalMaxPooling1D)<jupyter_output><empty_output><jupyter_text>Understanding 1D convolution for sequence dataRecall that in 2D convolutions, the layers extract 2D patches (areas) from image tensors. Similarly, a 1D convolution extracts 1D subsequences from a sequence.These 1D convolution layers can recognise local patterns in a sequence. Because the same input transformation is performed on every patch, a pattern learned in one position can later be recognised at a different position (translation invariance). A character-level 1D convnet is thus able to learn about morphology ("-ly", "-ing", "un-", "dis-", "-ed" etc.)  Implementing a 1D convnetIn Keras we can use the `Conv1D` layer, which takes in an input 3D tensors with shape `(samples, timesteps, features)` and outputs similarly shaped 3D tensors.The network consist of a stack of `Conv1D` and `MaxPooling1D` layers, similar to their 2D counterparts. You can affort to use a larger convolution windows with 1D convnets. A 3X3 layer has 9 features in a 2D convnet but a 3-dim layer in a 1D convnet has only 3 features.<jupyter_code># Ingestion
###########
VOCABULARY_SIZE = 10000
DOCUMENT_MAX_LENGTH = 500
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=VOCABULARY_SIZE)
x_train = sequence.pad_sequences(train_data, maxlen=DOCUMENT_MAX_LENGTH)
x_test = sequence.pad_sequences(test_data, maxlen=DOCUMENT_MAX_LENGTH)<jupyter_output><empty_output><jupyter_text>v1: Simple 1D Convnet<jupyter_code># Instantiate Model
###################
# clear_session()
# model1 = Sequential()
# model1.add(Embedding(VOCABULARY_SIZE, 128, input_length=DOCUMENT_MAX_LENGTH))
# model1.add(Conv1D(32, 7, activation='relu'))
# model1.add(MaxPooling1D(5))
# model1.add(Conv1D(32, 7, activation='relu'))
# model1.add(GlobalMaxPooling1D())
# model1.add(Dense(1))

# Compile
#########
# model1.compile(optimizer=RMSprop(learning_rate=1e-4), 
#                loss='binary_crossentropy',
#                metrics=['acc'])
# print(model1.summary())
# history1 = model1.fit(x_train, train_labels,
#                       epochs=10, 
#                       validation_split=0.2)
# Save Model & Training Results
###############################
model1.save('model64_1D_CNN.h5')
pd.DataFrame(history1.history).to_csv('model64_1D_CNN_history.csv', index=False)
metrics_df = pd.DataFrame(history1.history)
metrics_df['epoch'] = metrics_df.index+1
display(metrics_df.tail())

fig = plt.figure(figsize=(17,10))
ax1, ax2 = fig.add_subplot(2,1,1), fig.add_subplot(2,1,2)
metrics_df.plot(kind='scatter', x='epoch', y='loss', ax=ax1, label='train', color='blue')
metrics_df.plot(kind='line', x='epoch', y='val_loss', ax=ax1, label='validation', color='red',)
ax1.set_ylabel("Loss")
ax1.grid('GAINSBORO')
ax1.legend([])
ax1.set_xticks(range(0,11))

metrics_df.plot(kind='scatter', x='epoch', y='acc', ax=ax2, label='train', color='blue')
metrics_df.plot(kind='line', x='epoch', y='val_acc', ax=ax2, label='validation', color='red',)
ax2.set_ylabel("Accuracy")
ax2.grid('GAINSBORO')
ax2.set_xticks(range(0,11))
ax2.legend([])
plt.show()<jupyter_output><empty_output><jupyter_text>Combining CNNs and RNNs to process long sequencesBecause 1D convnets process input batches independently, they aren't sensitive to the order of the timesteps, unlike RNNs. To recognise longer-term patterns, you may stack many convolutional & pooling layers, resulting in upper layers that will see long chungs of the original inputs. We can see this when we use 1D convnets on the **temperature-forecasting** problem.<jupyter_code># Instantiate Model
###################
clear_session()
model2 = Sequential()
model2.add(Embedding(VOCABULARY_SIZE, 128, input_length=DOCUMENT_MAX_LENGTH))
model2.add(Conv1D(32, 7, activation='relu'))
model2.add(MaxPooling1D(5))
model2.add(Conv1D(32, 7, activation='relu'))
model2.add(GlobalMaxPooling1D())
model2.add(Dense(1))

Compile
########
model2.compile(optimizer=RMSprop(learning_rate=1e-4), 
               loss='binary_crossentropy',
               metrics=['acc'])
print(model2.summary())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12126436781609196, 'fraction_numerical': 0.026628352490421454, 'mean_word_length': 6.343178621659634, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '362361', 'license': 'MIT', 'n_tokens_mistral': 1731, 'n_tokens_neox': 1621, 'n_words': 469, 'path': 'chap06/textbook-chap-6-4.ipynb', 'repo_name': 'bryanblackbee/topic__deep-learning-python', 'size': '5220'}