<jupyter_start><jupyter_text>CS4618: Artificial Intelligence IOverfitting with Neural Networks    <NAME>    School of Computer Science and Information Technology    University College Cork Initialization$\newcommand{\Set}[1]{\{1\}}$ $\newcommand{\Tuple}[1]{\langle1\rangle}$ $\newcommand{\v}[1]{\pmb{1}}$ $\newcommand{\cv}[1]{\begin{bmatrix}1\end{bmatrix}}$ $\newcommand{\rv}[1]{[1]}$ $\DeclareMathOperator{\argmax}{arg\,max}$ $\DeclareMathOperator{\argmin}{arg\,min}$ $\DeclareMathOperator{\dist}{dist}$$\DeclareMathOperator{\abs}{abs}$<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt

from tensorflow.keras.datasets import mnist

from tensorflow.keras import Model
from tensorflow.keras import Input
from tensorflow.keras.layers.experimental.preprocessing import Rescaling
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Dropout

from tensorflow.keras.optimizers import SGD

from tensorflow.keras.regularizers import l1
from tensorflow.keras.regularizers import l2

from tensorflow.keras.callbacks import EarlyStopping
# MNIST dataset

# Load MNIST into four Numpy arrays
(mnist_x_train, mnist_y_train), (mnist_x_test, mnist_y_test) = mnist.load_data()
mnist_x_train = mnist_x_train.reshape((60000, 28 * 28))
mnist_x_test = mnist_x_test.reshape((10000, 28 * 28))<jupyter_output><empty_output><jupyter_text>Acknowledgement    The analogy between dropout and a company whose employees are told to toss a coin to        decide whether to go to work each morning comes from        <NAME>&eacute;ron: Hands-On Machine Learning with Scikit-Learn, Keras &amp;        TensorFlow (2nd edn), O'Reilly, 2019     Introduction    One of the central problems of deep learning is overfitting.    Reminder. If your model overfits, your main options are:                    gather more training examples;            remove noise in the training examples;            change model: move to a less complex model;            simplify by reducing the number of features;            stick with your existing model but                add constraints (if you can) to reduce its complexity.                            Here we'll look at                    reducing the network's size &mdash; an example of moving to a less complex model;            weight regularization &mdash; an example of adding constraints to reduce complexity;            dropout &mdash; also an example of adding constraints to reduce complexity; and            early stopping &mdash; a somewhat different way of avoiding overfitting.<jupyter_code># A network that overfits (a little!)

inputs = Input(shape=(28 * 28,))
x = Rescaling(scale=1./255)(inputs)
x = Dense(1024, activation="relu")(x)
x = Dense(1024, activation="relu")(x)
outputs = Dense(10, activation="softmax")(x)
overfitting_model = Model(inputs, outputs)
overfitting_model.compile(optimizer=SGD(learning_rate=0.01), loss="sparse_categorical_crossentropy")
history = overfitting_model.fit(mnist_x_train, mnist_y_train, epochs=20, batch_size=32, 
                                verbose=0, validation_split=0.20)
history.history["loss"][-1], history.history["val_loss"][-1]
# Training curve
pd.DataFrame(history.history).plot()<jupyter_output><empty_output><jupyter_text>Reducing Network Size    We can make the model (neural network) less complex by reducing the number of parameters.    Obviously enough, this is achieved by:                    reducing the number of hidden layers, and/or            reducing the number of neurons within the hidden layers.             Example of reducing network size<jupyter_code># Smaller network

inputs = Input(shape=(28 * 28,))
x = Rescaling(scale=1./255)(inputs)
x = Dense(256, activation="relu")(x)
outputs = Dense(10, activation="softmax")(x)
smaller_model = Model(inputs, outputs)
smaller_model.compile(optimizer=SGD(learning_rate=0.01), loss="sparse_categorical_crossentropy")
history = smaller_model.fit(mnist_x_train, mnist_y_train, epochs=20, batch_size=32, verbose=0, validation_split=0.2)
history.history["loss"][-1], history.history["val_loss"][-1]<jupyter_output><empty_output><jupyter_text>Weight Regularization    For linear regression, we used regularization to ensure that the coefficients $\v{\beta}$ took only        small values by penalizing large values in the loss function.                    Lasso: we penalized by the $\cal{l}_1$-norm (the sum of their absolute values).                        Ridge: we penalized by the $\cal{l}_2$-norm (the sum of their squares).                        A hyperparameter $\lambda$, called the 'regularization parameter' controlled the balance                 between fitting the data versus shrinking the parameters.                            Weight Regularization in neural networks is the same idea, but applied to the weights         in the layers of a network.<jupyter_code># Regularized network

inputs = Input(shape=(28 * 28,))
x = Rescaling(scale=1./255)(inputs)
x = Dense(1024, activation="relu", kernel_regularizer=l2(0.0001))(x)
x = Dense(1024, activation="relu", kernel_regularizer=l2(0.0001))(x)
outputs = Dense(10, activation="softmax", kernel_regularizer=l2(0.0001))(x)
regularized_model = Model(inputs, outputs)
regularized_model.compile(optimizer=SGD(learning_rate=0.01), loss="sparse_categorical_crossentropy")
history = regularized_model.fit(mnist_x_train, mnist_y_train, epochs=20, batch_size=32, verbose=0, validation_split=0.2)
history.history["loss"][-1], history.history["val_loss"][-1]<jupyter_output><empty_output><jupyter_text>Dropout    Imagine we have a layer that uses dropout with droput rate $p$, e.g. $p=0.5$.    Then, in a given step of the backprop algorithm, each neuron in the layer has        probability $p$ of being ignored &mdash; treated as if it were not there.                                 One way of doing dropout    Suppose the droppout rate is $p$.    Training. For any given mini-batch:                    In the forward propagation,                                     decide which neurons will be dropped (chosen with probability $p$);                    set the activations of the dropped neurons to zero;                    multiply the activations of the kept neurons by $1 / (1 - p)$.                                        In the backpropagation, ignore the dropped out neurons.                Note that different neurons will get dropped for each mini-batch.        Testing. No change.    But why did we multiply activations by $1/ (1 - p)$?                    In testing, for $p=0.5$ a neuron in the next layer will receive input from on average                twice as many neurons as it did in training.                        The multiplication by $1/(1 - p)$ compensates for this.                         Why does dropout reduce overfitting?    Consider a company whose employees were told to toss a coin every monring to decide        whether to go to work or not.                    The organization would need to become more resilient. It could not rely on any                one employee to perform critical tasks: the expertise would need to be spread                across many employees. They must become more like generalists, less like specialists.                            Similarly, in dropout layers, neurons learn more robust features.             Another way to think about it.                    Since a neuron can be present or absent, it's like training on a different neural                network at each step.                        The final result is  a bit like an ensemble of these many different virtual                neural networks.                            However, it typically increases the number of epochs needed for convergence        (roughly double when $p=0.5$).     Dropout in Keras<jupyter_code># Network with dropout

inputs = Input(shape=(28 * 28,))
x = Rescaling(scale=1./255)(inputs)
x = Dense(1024, activation="relu")(x)
x = Dropout(0.5)(x)
x = Dense(1024, activation="relu")(x)
x = Dropout(0.5)(x)
outputs = Dense(10, activation="softmax")(x)
model_with_dropout = Model(inputs, outputs)
model_with_dropout.compile(optimizer=SGD(learning_rate=0.01), loss="sparse_categorical_crossentropy")
history = model_with_dropout.fit(mnist_x_train, mnist_y_train, epochs=20, batch_size=32, verbose=0, validation_split=0.2)
history.history["loss"][-1], history.history["val_loss"][-1]<jupyter_output><empty_output><jupyter_text>Early Stopping    We know that a sign of overfitting is that validation error stops getting lower and starts        getting larger.        We can exploit this during Gradient Descent as an other way of avoiding overfitting,        known as early stopping:                    During Gradient Descent, monitor validation error (or loss).            Interrupt training when the validation error has stopped improving for a certain number                of epochs.                         Early stopping in Keras    In Keras, this is done using the EarlyStopping callback.    The patience argument allows you to specify how many epochs must pass with        no improvement relative to the current best.        restore_best_weights=True restores the weights and biases from when validation        error was at its lowest.<jupyter_code>history = overfitting_model.fit(mnist_x_train, mnist_y_train, epochs=20, batch_size=32, 
                                verbose=0, validation_split=0.2, 
                                callbacks=[EarlyStopping(monitor="val_loss", patience=2, restore_best_weights=True)])
[(l, v) for l, v in zip(history.history["loss"], history.history["val_loss"])]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08627654065251165, 'fraction_numerical': 0.020714655618850338, 'mean_word_length': 3.3184257602862255, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '413097', 'license': 'MIT', 'n_tokens_mistral': 2753, 'n_tokens_neox': 2574, 'n_words': 926, 'path': 'ai1/lectures/AI1_20_overfitting_with_neural_networks.ipynb', 'repo_name': 'derekgbridge/artificial_intelligence', 'size': '9659'}