<jupyter_start><jupyter_text>Deep LearningPractical 3b - Overfitting and UnderfittingAY2020/21 Semester<jupyter_code>from tensorflow import keras
print('keras: ', keras.__version__)
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))<jupyter_output><empty_output><jupyter_text>ObjectivesAfter completing this practical exercise, students should be able to:1. [Understand how to prevent overfitting in Neural Networks](over)2. [Exercise - Develop your own model and try to avoid overfitting](exc)  Load Data Note: in this notebook we will be using the IMDB test set as our validation set. It doesn't matter in this context.Let's prepare the data using the code from Practical 2:<jupyter_code>from tensorflow.keras.datasets import imdb
import numpy as np

(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)

def vectorize_sequences(sequences, dimension=10000):
    # Create an all-zero matrix of shape (len(sequences), dimension)
    results = np.zeros((len(sequences), dimension))
    for i, sequence in enumerate(sequences):
        results[i, sequence] = 1.  # set specific indices of results[i] to 1s
    return results

# Our vectorized training data
x_train = vectorize_sequences(train_data)
# Our vectorized test data
x_test = vectorize_sequences(test_data)
# Our vectorized labels
y_train = np.asarray(train_labels).astype('float32')
y_test = np.asarray(test_labels).astype('float32')<jupyter_output><empty_output><jupyter_text>1. How to prevent overfitting in Neural Networks   1.1 Reducing the network's size Our original network was as such:<jupyter_code>from tensorflow.keras import models
from tensorflow.keras import layers

original_model = models.Sequential()
original_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
original_model.add(layers.Dense(16, activation='relu'))
original_model.add(layers.Dense(1, activation='sigmoid'))

original_model.compile(optimizer='rmsprop',
                       loss='binary_crossentropy',
                       metrics=['acc'])<jupyter_output><empty_output><jupyter_text>Now let's try to replace it with this smaller network:<jupyter_code>smaller_model = models.Sequential()
smaller_model.add(layers.Dense(4, activation='relu', input_shape=(10000,)))
smaller_model.add(layers.Dense(4, activation='relu'))
smaller_model.add(layers.Dense(1, activation='sigmoid'))

smaller_model.compile(optimizer='rmsprop',
                      loss='binary_crossentropy',
                      metrics=['acc'])
original_hist = original_model.fit(x_train, y_train,
                                   epochs=20,
                                   batch_size=512,
                                   validation_data=(x_test, y_test))
smaller_model_hist = smaller_model.fit(x_train, y_train,
                                       epochs=20,
                                       batch_size=512,
                                       validation_data=(x_test, y_test))
epochs = range(1, 21)
original_val_loss = original_hist.history['val_loss']
smaller_model_val_loss = smaller_model_hist.history['val_loss']<jupyter_output><empty_output><jupyter_text>Here's a comparison of the validation losses of the original network and the smaller network. The dots are the validation loss values of the smaller network, and the crosses are the initial network (remember: a lower validation loss signals a better model).<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

# b+ is for "blue cross"
plt.plot(epochs, original_val_loss, 'b+', label='Original model')
# "bo" is for "blue dot"
plt.plot(epochs, smaller_model_val_loss, 'bo', label='Smaller model')
plt.xlabel('Epochs')
plt.ylabel('Validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>As you can see, the smaller network starts overfitting later than the reference one (after 6 epochs rather than 4) and its performance degrades much more slowly once it starts overfitting.Now, for kicks, let's add to this benchmark a network that has much more capacity, far more than the problem would warrant:<jupyter_code>bigger_model = models.Sequential()
bigger_model.add(layers.Dense(512, activation='relu', input_shape=(10000,)))
bigger_model.add(layers.Dense(512, activation='relu'))
bigger_model.add(layers.Dense(1, activation='sigmoid'))

bigger_model.compile(optimizer='rmsprop',
                     loss='binary_crossentropy',
                     metrics=['acc'])
bigger_model_hist = bigger_model.fit(x_train, y_train,
                                     epochs=20,
                                     batch_size=512,
                                     validation_data=(x_test, y_test))<jupyter_output>Train on 25000 samples, validate on 25000 samples
Epoch 1/20
25000/25000 [==============================] - 3s 136us/sample - loss: 0.4540 - acc: 0.8048 - val_loss: 0.3156 - val_acc: 0.8700
Epoch 2/20
25000/25000 [==============================] - 3s 111us/sample - loss: 0.2152 - acc: 0.9151 - val_loss: 0.2851 - val_acc: 0.8861
Epoch 3/20
25000/25000 [==============================] - 3s 113us/sample - loss: 0.1211 - acc: 0.9566 - val_loss: 0.3766 - val_acc: 0.8795
Epoch 4/20
25000/25000 [==============================] - 3s 112us/sample - loss: 0.0619 - acc: 0.9812 - val_loss: 0.4692 - val_acc: 0.8771
Epoch 5/20
25000/25000 [==============================] - 3s 112us/sample - loss: 0.0967 - acc: 0.9860 - val_loss: 0.4731 - val_acc: 0.8826
Epoch 6/20
25000/25000 [==============================] - 3s 112us/sample - loss: 0.0021 - acc: 0.9998 - val_loss: 0.6848 - val_acc: 0.8807
Epoch 7/20
25000/25000 [==============================] - 3s 112us/sample - loss: 0.0064 - acc: 0.9982 - val_l[...]<jupyter_text>Here's how the bigger network fares compared to the reference one. The dots are the validation loss values of the bigger network, and the crosses are the initial network.<jupyter_code>bigger_model_val_loss = bigger_model_hist.history['val_loss']

plt.plot(epochs, original_val_loss, 'b+', label='Original model')
plt.plot(epochs, bigger_model_val_loss, 'bo', label='Bigger model')
plt.xlabel('Epochs')
plt.ylabel('Validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>The bigger network starts overfitting almost right away, after just one epoch, and overfits much more severely. Its validation loss is also more noisy.Meanwhile, here are the training losses for our two networks:<jupyter_code>original_train_loss = original_hist.history['loss']
bigger_model_train_loss = bigger_model_hist.history['loss']

plt.plot(epochs, original_train_loss, 'b+', label='Original model')
plt.plot(epochs, bigger_model_train_loss, 'bo', label='Bigger model')
plt.xlabel('Epochs')
plt.ylabel('Training loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>As you can see, the bigger network gets its training loss near zero very quickly. The more capacity the network has, the quicker it will be able to model the training data (resulting in a low training loss), but the more susceptible it is to overfitting (resulting in a large difference between the training and validation loss).  1.2 Adding weight regularizationIn Keras, weight regularization is added by passing _weight regularizer instances_ to layers as keyword arguments. Let's add L2 weight regularization to our movie review classification network:<jupyter_code>from tensorflow.keras import regularizers

l2_model = models.Sequential()
l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),
                          activation='relu', input_shape=(10000,)))
l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),
                          activation='relu'))
l2_model.add(layers.Dense(1, activation='sigmoid'))
l2_model.compile(optimizer='rmsprop',
                 loss='binary_crossentropy',
                 metrics=['acc'])<jupyter_output><empty_output><jupyter_text>`l2(0.001)` means that every coefficient in the weight matrix of the layer will add `0.001 * weight_coefficient_value` to the total loss of the network. Here's the impact of our L2 regularization penalty:<jupyter_code>l2_model_hist = l2_model.fit(x_train, y_train,
                             epochs=20,
                             batch_size=512,
                             validation_data=(x_test, y_test))
l2_model_val_loss = l2_model_hist.history['val_loss']

plt.plot(epochs, original_val_loss, 'b+', label='Original model')
plt.plot(epochs, l2_model_val_loss, 'bo', label='L2-regularized model')
plt.xlabel('Epochs')
plt.ylabel('Validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>As you can see, the model with L2 regularization (dots) has become much more resistant to overfitting than the reference model (crosses), even though both models have the same number of parameters.As alternatives to L2 regularization, you could use one of the following Keras weight regularizers:<jupyter_code>from tensorflow.keras import regularizers

# L1 regularization
regularizers.l1(0.001)

# L1 and L2 regularization at the same time
regularizers.l1_l2(l1=0.001, l2=0.001)<jupyter_output><empty_output><jupyter_text>1.3 Adding dropoutIn Keras you can introduce dropout in a network via the `Dropout` layer, which gets applied to the output of layer right before it. Let's add two `Dropout` layers in our IMDB network to see how well they do at reducing overfitting:<jupyter_code>dpt_model = models.Sequential()
dpt_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
dpt_model.add(layers.Dropout(0.5))
dpt_model.add(layers.Dense(16, activation='relu'))
dpt_model.add(layers.Dropout(0.5))
dpt_model.add(layers.Dense(1, activation='sigmoid'))

dpt_model.compile(optimizer='rmsprop',
                  loss='binary_crossentropy',
                  metrics=['acc'])
dpt_model_hist = dpt_model.fit(x_train, y_train,
                               epochs=20,
                               batch_size=512,
                               validation_data=(x_test, y_test))<jupyter_output>Train on 25000 samples, validate on 25000 samples
Epoch 1/20
25000/25000 [==============================] - 3s 108us/sample - loss: 0.6254 - acc: 0.6417 - val_loss: 0.5041 - val_acc: 0.8556
Epoch 2/20
25000/25000 [==============================] - 2s 81us/sample - loss: 0.4907 - acc: 0.7758 - val_loss: 0.3836 - val_acc: 0.8763
Epoch 3/20
25000/25000 [==============================] - 2s 80us/sample - loss: 0.4006 - acc: 0.8413 - val_loss: 0.3155 - val_acc: 0.8864
Epoch 4/20
25000/25000 [==============================] - 2s 82us/sample - loss: 0.3400 - acc: 0.8725 - val_loss: 0.2850 - val_acc: 0.8880
Epoch 5/20
25000/25000 [==============================] - 2s 79us/sample - loss: 0.2921 - acc: 0.8930 - val_loss: 0.2807 - val_acc: 0.8868
Epoch 6/20
25000/25000 [==============================] - 2s 81us/sample - loss: 0.2624 - acc: 0.9050 - val_loss: 0.2860 - val_acc: 0.8887
Epoch 7/20
25000/25000 [==============================] - 2s 80us/sample - loss: 0.2370 - acc: 0.9162 - val_loss: 0[...]<jupyter_text>Let's plot the results:<jupyter_code>dpt_model_val_loss = dpt_model_hist.history['val_loss']

plt.plot(epochs, original_val_loss, 'b+', label='Original model')
plt.plot(epochs, dpt_model_val_loss, 'bo', label='Dropout-regularized model')
plt.xlabel('Epochs')
plt.ylabel('Validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>Again, a clear improvement over the reference network.To recap: here the most common ways to prevent overfitting in neural networks:* Getting more training data.* Reducing the capacity of the network.* Adding weight regularization.* Adding dropout.  2. Exercise Please develop your own model for IMDB review classification problem following "The Universal Workflow of Machine Learning" and try to maximize the validation accuracy. When you are regularizing the model you can utilize the below tecniques:* Add dropout* Try different architectures: add or remove layers, increase or decrease the number of units per layer* Add L1 and / or L2 regularization* Adjust the hyperparameters, e.g. the learning rate of the optimizer, the number of epochs, batch_size and etc.Provide your codes and comments in the below boxes.<jupyter_code># Task 1: Build and Compile the model. 
# Remember to use the above mentioned regularization techniques
from tensorflow.keras.datasets import imdb
import numpy as np
from tensorflow.keras import models
from tensorflow.keras import layers

(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)

def vectorize_sequences(sequences, dimension=10000):
    # Create an all-zero matrix of shape (len(sequences), dimension)
    results = np.zeros((len(sequences), dimension))
    for i, sequence in enumerate(sequences):
        results[i, sequence] = 1.  # set specific indices of results[i] to 1s
    return results

# Our vectorized training data
x_train = vectorize_sequences(train_data)
# Our vectorized test data
x_test = vectorize_sequences(test_data)
# Our vectorized labels
y_train = np.asarray(train_labels).astype('float32')
y_test = np.asarray(test_labels).astype('float32')
#normal
model = models.Sequential()
model.add(layers.Dense(8, activation='relu', input_shape=(10000,)))
model.add(layers.Dense(8, activation='relu'))
model.add(layers.Dense(8, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(optimizer='rmsprop',
                       loss='binary_crossentropy',
                       metrics=['acc'])
#add dropout
dpt_model = models.Sequential()
dpt_model.add(layers.Dense(8, activation='relu', input_shape=(10000,)))
dpt_model.add(layers.Dropout(0.5))
dpt_model.add(layers.Dense(8, activation='relu'))
dpt_model.add(layers.Dropout(0.5))
dpt_model.add(layers.Dense(8, activation='relu'))
dpt_model.add(layers.Dropout(0.5))
dpt_model.add(layers.Dense(1, activation='sigmoid'))

dpt_model.compile(optimizer='rmsprop',
                  loss='binary_crossentropy',
                  metrics=['acc'])

from tensorflow.keras import regularizers
#L1
l1_model = models.Sequential()
l1_model.add(layers.Dense(8, kernel_regularizer=regularizers.l1(0.001),
                          activation='relu', input_shape=(10000,)))
l1_model.add(layers.Dense(8, kernel_regularizer=regularizers.l1(0.001),
                          activation='relu'))
l1_model.add(layers.Dense(8, kernel_regularizer=regularizers.l1(0.001),
                          activation='relu'))
l1_model.add(layers.Dense(1, activation='sigmoid'))

l1_model.compile(optimizer='rmsprop',
                 loss='binary_crossentropy',
                 metrics=['acc'])
#L2
l2_model = models.Sequential()
l2_model.add(layers.Dense(8, kernel_regularizer=regularizers.l2(0.001),
                          activation='relu', input_shape=(10000,)))
l2_model.add(layers.Dense(8, kernel_regularizer=regularizers.l2(0.001),
                          activation='relu'))
l2_model.add(layers.Dense(8, kernel_regularizer=regularizers.l2(0.001),
                          activation='relu'))
l2_model.add(layers.Dense(1, activation='sigmoid'))

l2_model.compile(optimizer='rmsprop',
                 loss='binary_crossentropy',
                 metrics=['acc'])
# Task 2: Compile and Fit the model
# You can adjust the number of epochs, the batch size, the learning rate of the optimizer and etc.
model_hist = model.fit(x_train, y_train,
                               epochs=50,
                               batch_size=256,
                               validation_data=(x_test, y_test))
dpt_model_hist = dpt_model.fit(x_train, y_train,
                               epochs=50,
                               batch_size=256,
                               validation_data=(x_test, y_test))
l1_model_hist = l1_model.fit(x_train, y_train,
                               epochs=50,
                               batch_size=256,
                               validation_data=(x_test, y_test))
l2_model_hist = l2_model.fit(x_train, y_train,
                               epochs=50,
                               batch_size=256,
                               validation_data=(x_test, y_test))
# Task 3: Plot the training and validation accuracy
import matplotlib.pyplot as plt
%matplotlib inline

epochs = range(1, 51)
model_train_accuracy = model_hist.history['acc']
dpt_model_train_accuracy = dpt_model_hist.history['acc']
l1_model_hist_train_accuracy = l1_model_hist.history['acc']
l2_model_hist_train_accuracy = l2_model_hist.history['acc']

model_val_accuracy = model_hist.history['val_acc']
dpt_model_val_accuracy = dpt_model_hist.history['val_acc']
l1_model_hist_val_accuracy = l1_model_hist.history['val_acc']
l2_model_hist_val_accuracy = l2_model_hist.history['val_acc']

plt.plot(epochs, model_train_accuracy, label='Original train model')
plt.plot(epochs, dpt_model_train_accuracy, label='DPT train model')

plt.plot(epochs, model_val_accuracy, label='Original val model')
plt.plot(epochs, dpt_model_val_accuracy, label='DPT val model')

plt.title('Oridinal vs DPT')
plt.xlabel('Epochs')
plt.ylabel('Train & Val accuracy')
plt.legend()

plt.show()

plt.plot(epochs, model_train_accuracy, label='Original train model')
plt.plot(epochs, l1_model_hist_train_accuracy, label='l1 train model')

plt.plot(epochs, model_val_accuracy, label='Original val model')
plt.plot(epochs, l1_model_hist_val_accuracy, label='l1 val model')

plt.title('Oridinal vs L1')
plt.xlabel('Epochs')
plt.ylabel('Train & Val accuracy')
plt.legend()

plt.show()

plt.plot(epochs, model_train_accuracy, label='Original train model')
plt.plot(epochs, l2_model_hist_train_accuracy, label='l2 train model')

plt.plot(epochs, model_val_accuracy, label='Original val model')
plt.plot(epochs, l2_model_hist_val_accuracy, label='l2 val model')

plt.title('Oridinal vs L2')
plt.xlabel('Epochs')
plt.ylabel('Train & Val accuracy')
plt.legend()

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13009665592711922, 'fraction_numerical': 0.04660593267414732, 'mean_word_length': 4.082721626199887, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '219678', 'license': 'MIT', 'n_tokens_mistral': 6313, 'n_tokens_neox': 5669, 'n_words': 1399, 'path': 'DL/week03/Practical_3b-Overfitting-and-Underfitting.ipynb', 'repo_name': 'dodieboy/Np_class', 'size': '18002'}